Commit Graph

1763 Commits

Author SHA1 Message Date
Michael D Kinney
50dc5f2f31 BaseTools/GetFw: Skip R_X86_64_NONE relocations
Building .aslc files in a module for host-based unit
testing in Linux/GCC X64 environments may introduce
relocations for a symbol in a shared library from
lcov or asan.

Update logic to not generate an error message and exit
if the symbol is unknown but the relocation type is
R_X86_64_NONE.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
2025-06-24 19:32:41 +00:00
Nate DeSimone
c0ef2b0178 BaseTools: Add support for mingw-w64
Adds support for building the C language BaseTools for Windows using
toolchains based on mingw-w64.

Mingw-w64 is a collection of header files, libraries, and tools that
when combined with a compiler enable development of Windows software.
Mingw-w64 is a fork of the original MinGW (Minimalist GNU for Windows).
Most active development on MinGW has ceased and mingw-w64 is now the
actively maintained successor.  Mingw-w64 provides a libc implementation
built on top of Microsoft's UCRT (Universal C Runtime) with all
nessesary compiler bindings needed to support the C++11 feature set.

Modern mingw-w64 development appears to have coalesced around MSYS2,
which produces a distributions of both GCC and LLVM/Clang that use
mingw-w64 to target the Windows OS. This MSYS2 Clang distribution has a
UNIX-like directory layout and includes Windows binaries of GNU Make.
Combined with the open source licensing, MSYS2's Clang distribution is a
highly attractive choice as an alternative Windows SDK for open source
projects such as TianoCore.

If one wishes to use EDK II to build UEFI firmware on the Windows
platform, then the C BaseTools need to be compiled as Windows
applications. This includes the PcdValueInit.exe program, which needs
to be recompiled every time a firmware build is run in order to
regenerate the initial values for structured PCDs. Currently, BaseTools
only supports the Visual C++ toolchain on the Windows platform. The
following new features have been added to enable usage of the toolchains
derived from mingw-w64:

 - Fixes to the BaseTools C source code to support the use of a
   GCC-style compiler on the Windows OS.
 - The GNU Make-style Makefiles for the C BaseTools have been modified
   to support Windows. Both GCC + mingw-w64 and Clang + mingw-w64 have
   been tested and confirmed to build a working BaseTools.
 - BaseTools now supports generating GNU Make-style Makefiles on the
   Windows platform for the purpose of building firmware.
 - edksetup.bat has been modified to optionally build BaseTools via
   mingw-w64. There is no impact to the existing support for Visual C++
   and Visual C++ remains the default toolchain.

Usage Instructions:

For the vast majority of users, the only system setup change nessesary
to use a mingw-w64 toolchain is to set the BASETOOLS_MINGW_PATH to the
directory containing the desired mingw-w64 based toolchain.

A new command line argument has been added to edksetup.bat: Mingw-w64
If this command line argument is set, then the script will set the
BASETOOLS_MINGW_BUILD environment variable. The user can also opt to set
this environment variable manually before running edksetup.bat

If BASETOOLS_MINGW_BUILD is defined, then the BASETOOLS_MINGW_PATH
environment variable must point to the directory containing the
mingw-w64 toolchain.

If CLANG_BIN is not defined and %BASETOOLS_MINGW_PATH%\bin\clang.exe
exists, then edksetup.bat will set CLANG_BIN=%BASETOOLS_MINGW_PATH%\bin\
This removes the requirement to configure the CLANG_BIN environment
variable manually in order to run a CLANGPDB or CLANGDWARF build if one
has the MSYS2 Clang distribution installed. If one wishes to use a
different copy of Clang (for example official LLVM binaries) to build
firmware and only use the MSYS2 Clang to build BaseTools, then one can
continue to set the CLANG_BIN environment variable, same as before. I
have tested the MSYS2 Clang distribution against the official LLVM
distribution and can confirm that if the compiler version is the same
the emitted machine code is identical between the two. Interestingly,
the MSYS2 Clang distribution emits the path to the PDB file using "/" as
the path seperator instead of "\". That appears to be the only
difference in output. Therefore, using the MSYS2 Clang distribution to
compile firmware seems a reasonable choice.

If CLANG_HOST_BIN is not defined and BASETOOLS_MINGW_BUILD is defined
and %BASETOOLS_MINGW_PATH%\bin\mingw32-make.exe exists, then
edksetup.bat will add %BASETOOLS_MINGW_PATH%\bin\ to the PATH and set
CLANG_HOST_BIN=mingw32-
This enable usage of the GNU Make included in the mingw-w64 toolchain
to build firmware in addition to BaseTools. if BASETOOLS_MINGW_BUILD is
not defined, edksetup.bat will continue to set CLANG_HOST_BIN=n, which
uses nmake to build firmware. This behavior can be overridden by
manually setting the value of CLANG_HOST_BIN before executing
edksetup.bat if one wishes to use a specific Make utility for the
CLANGPDB/CLANGDWARF toolchains.

References:
 - https://www.mingw-w64.org/
 - https://www.msys2.org/

Co-authored-by: Sandesh Jain <sandesh.jain@intel.com>
Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
2025-06-07 17:56:04 +00:00
Zhiguang Liu
9b6d1126de BaseTools: Add GUID section for build report
Add GUID section for build report.
Also, change the GUID format to string format to be easier to parse

Signed-off-by: Zhiguang Liu <zhiguang.liu@intel.com>
2025-05-28 06:00:44 +00:00
Rebecca Cran
a78f525634 BaseTools: Update CParser4 to import TextIO from typing
In Python 3.13 typing.io no longer exists.
Instead, import TextIO from typing.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
2025-05-26 17:09:27 -06:00
Nate DeSimone
43e29830ef BaseTools: Fix Macro Expansion on Machine Architecture for Components
BaseTools currently does not expand macros for component architecture when
nested !include directives are used. This breaks use cases like
[Component.$(DXE_ARCH)].

The fix is to add explicit macro expansion when computing the Arch value
for component lines in DscParser.

Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
2025-05-09 09:01:43 +08:00
Nate DeSimone
bf99048365 BaseTools: Add support for GCC preprocessor line directives
This change adds support for GCC-style preprocessor line
directives as documented in:
https://gcc.gnu.org/onlinedocs/cpp/Preprocessor-Output.html

On Windows systems, one can use line-markers to see  which .vfr
file was used to generate a *.i file in the Build directory.
This is useful for debugging VFR compilation failures.

With this change, the VfrCompiler will not generate compilation
errors if the *.i file contains GCC-style line-markers. Without
this change, one must disable the pre-processor's emission of
line-markers, removing the debug aid they provide.

Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
2025-04-28 22:59:30 +00:00
Doug Flick
0f1c0d2d2d BaseTools:Expression.py Adding a check for missing '}' in the parser
This check is to catch cases where a missing '}' exists in a dec or dsc
file.

Signed-off-by: Doug Flick <dougflick@microsoft.com>
2025-04-14 20:53:02 +00:00
Doug Flick
1805068b5e BaseTools:Expression.py Add Assertion for PCD with value {}
This adds an assertion to the PCD class in the Expression.py
file to check for zero-byte PCDs.

Signed-off-by: Doug Flick <dougflick@microsoft.com>
2025-04-14 20:53:02 +00:00
Doug Flick
a1b623b938 BaseTools:Expression.py Size used before Init
The following example fails to be parsed correctly due to Size
being used in the outer scope but initialized in the inner
scope

```
gPlatformPkgTokenSpaceGuid.PcdSecureBootDbxBinaryFile|{}
```

Problematic code:

```python
for Item in NewPcdValueList:
      Size = 0
      # ....

if Size > 0:
      PcdValue = '{' + ', '.join(AllPcdValueList) + '}'
````

Signed-off-by: Doug Flick <dougflick@microsoft.com>
2025-04-14 20:53:02 +00:00
Yuwei Chen
7c6ef8eac2 BaseTools: Fix FMMT Compress wrong behavior
When using the replace function, if the replaced file
located in a compress section, then the tool will re-build
the section. As the compress status set in the wrong place,
in some situation, the compress will do twice times which is
a wrong behavior.

This patch is used to fix this issue.

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Ashraf Ali S <ashraf.ali.s@intel.com>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
2025-04-11 10:40:17 +08:00
Ray Ni
d0b64b21a3 BaseTools: Dump library dependency chain on build failure
When a module M depends on L1, which depends on L2,
which depends on L3, the build fails when the library instance
of L3 cannot be found according to the library class-instance
mapping configuration specified in the DSC file.
When such failure happens, the build tool only prints that the
instance of L3 required by module M cannot be found. But it
does not tell how L3 is required by M.

The change enhances build tool to print the entire dependency
chain when such failure happens.
With the change, the new error message will be as follows:

<dsc-path>(...): error 4000: Instance of library class [L3] is not
found for module [M], [L3] is:
  consumed by <instance of L2>
    consumed by <instance of L1>

Signed-off-by: Ray Ni <ray.ni@intel.com>
2025-04-08 08:40:49 +00:00
Michael D Kinney
09faa4a1ce BaseTools/build: Reduce special FV full to warning
If an FV_SPARE_SPACE_THRESHOLD is enabled and an FV
is 100% full with 0 bytes free, then this is likely
a special FV that may have alignment requirements
for the FFS file for both the start address and the
length and the FFS file consumes all the available
FV space.

Reduce FV_SPARE_SPACE_THRESHOLD from an error to a
warning if this FV 0 bytes free condition is
detected.

PR #10828 introduced the generation of these error
conditions for an FV with large alignment requirements.
The pad region before the aligned FFS file used to
be counted as free space even though it could never
be used due to the alignment requirements. There was
actually no free space available. PR #10828 fixed the
free space calculation to properly show it as 0 bytes
free, and this change then caused build error when
FV_SPARE_SPACE_THRESHOLD feature was enabled. The
reduction to a warning for this condition allows the
build to complete with errors and also provides a
build log warning message for review.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
2025-03-31 16:04:37 +00:00
Jason1 Lin
a7ab45ace2 BaseTools/FMMT: Fix GUID Tool Not Found the Shell Script Issue
- FMMT tool would use the "PATH" environment variable for locating
  the required GUID tool.

- On Windows-like system, batch file not found in the "PATH" environment
  variable when "shell=False".

- This issue required commands to include program extensions or
  absolute paths.

- This patch sets "shell=True" to extend the support for batch files,
  including scripts in BinWrappers under BaseTools.

- Converted input commands from lists to strings to ensure proper
  argument interpretation in POSIX-like shell scripts.

Signed-off-by: Jason1 Lin <jason1.lin@intel.com>
2025-03-26 05:37:32 +00:00
Zhiguang Liu
e01f4180b3 BaseTools: Fix missing module PCD in compile information of build report
Module PCD in compile information is missed in module_report.json

Signed-off-by: Zhiguang Liu <zhiguang.liu@intel.com>
2025-03-25 10:10:20 +00:00
Phil Noh
b279ac9f36 BaseTools/GenFv: Ensure the minimum pad file size for the FV with VTF
In case of the FV with VTF, the left size should be enough to add the
minimum pad file size (EFI_FFS_FILE_HEADER, 0x18). It prevents the build
error, "GenFv: ERROR 0006: invalid FFS file header checksum" caused by the
pad file overwriting some header data in VTF. This includes these updates
for CalculateFvSize() function.

1. If NumBlocks is not defined, ensure the minimum pad file size for the
left size (if the pad file is required as VTF is not bottom aligned at end
of block, insert EFI_FFS_FILE_HEADER to ensure the pad file size)
2. If NumBlocks is defined, report more clear error message (the required
fv image size = 0x%x. the set fv image size = 0x%x. Free space left is not
enough to add a pad file (0x18))
3. Remove MaxPadFileSize, which is reported when the taken size is same as
the total size. It can not be the actual left size to add an FFS file. It
causes confusion when referring to the build log (FV Space Information)

Signed-off-by: Phil Noh <Phil.Noh@amd.com>
2025-03-22 01:58:40 +00:00
Michael D Kinney
03e66e0a03 BaseTools/Source/C: Disable VS flexible array member warning
Add /wd4200 to all visual studio builds of C tools in BaseTools.
This disables warnings for use of flexible array members that
are allowed in edk2 include files. Some tools use include files
from MdePkg that use flexible array members.

This matches the warning disables used to build structured PCD in
BaseTools/Source/Python/Workspace/DscBuildData.py where flexible
array members are more widely used.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
2025-03-12 11:16:55 +08:00
Rebecca Cran
259e1f04c6 BaseTools: Add @verbatim as allowed Doxygen tag
Per the discussion in https://github.com/tianocore/edk2/pull/6476,
update Ecc code to allow the @verbatim Doxygen tag.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
2025-02-04 00:12:00 +00:00
Michael D Kinney
62b0698309 BaseTools/AutoGen: GenMake response file quotes strings
If command line options are moved into a response file
of a GCC family build, then the file path separators are
converted from '\' to '/'. However, this can corrupt
command line options that are quoted strings.

Update GenMake to no convert '\' to '/' in quoted strings.

Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
2025-02-02 19:22:42 +00:00
Gerd Hoffmann
e063f8b8a5 BaseTools/Pccts: set C standard
The prehistoric code base doesn't build with ISO C23.  Set the C
standard to C11 (for both clang and gcc) so it continues to build with
gcc 15 (which uses C23 by default).

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2025-01-31 05:23:39 +00:00
Ashraf Ali S
6d2143f685 BaseTools: Fix NoneType parent reference in FMMT operations
This patch addresses an issue in the FMMT operations where the parent
reference is not checked for NoneType. This oversight can lead to an
AttributeError: 'NoneType' object has no attribute 'Name' when
attempting to access the parent reference. The fix involves adding a
check for NoneType before accessing the parent reference to ensure that
the operations handle such cases gracefully.

The affected functions include:
- AddNewFfs
- ReplaceFfs
- ExtractFfs

These functions now include proper checks to prevent the AttributeError.

Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
2025-01-27 16:26:18 +00:00
Yang Gang
3ac092cf72 BaseTools: Clean up os.path.normcase and os.path.normpath usage
Refer to the docs of python, `os.path.normcase(path)` function:
"Normalize the case of a pathname. On Windows, convert all characters in
the pathname to lowercase, and also convert forward slashes to backward
slashes. On other operating systems, return the path unchanged."

`os.path.normpath(path)` also convert forward slashes to backward slashes.

So call `os.path.normcase` after `os.path.normpath` just convert path to
lowercase on Windows(only).

And Windows is case-insensitive but case-preserving.

So the usage of `os.path.normcase(os.path.normpath(path))` can be
simplified to `os.path.normpath(path)`. Then we can use case-preserving
paths rather than lowercase paths in compile_commands.json file
or build log.

But this patch continue to use `os.path.normcase`
when comparing/searching paths.

Signed-off-by: Yang Gang <yanggang@byosoft.com.cn>
2025-01-09 07:25:45 +00:00
Ashraf Ali
1cc78814cd Remove dependancy on re.T
Signed-off-by: Ashraf Ali <ashraf.ali.s@intel.com>
2024-12-24 07:32:42 +00:00
kuqin12
3c8016b302 BaseTools: Support custom library build for base tools on Linux ARM
This change added the build script to cross compile the base tool
binaries for Linux ARM/AARCH64 systems.

The needed libuuid system library is pulled from source file and rebuilt
to support the corresponding library dependencies. Individual tools'
makefiles are also updated to link the cross compiled library as well.

The EDK2 base tool build script was also updated to support such change.

This was tested functional on Linux ARM host system.

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Kun Qin <kun.qin@microsoft.com>
2024-12-20 22:57:04 +00:00
Kun Qin
74bf7f55c1 BaseTools: Adding cross compilation of BaseTool for Windows ARM/ARM64
This change adds the support of crossbuilding basetool for Windows ARM/
ARM64 systems, which will enable the generally available pipeline agents
to build binary tools and make releases as they see fit.

The EDK2 base tools build script is also updated to support cross
compilation using this script.

The crossbuilt binary output is tested on Windows ARM based hardware
systems.

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Kun Qin <kun.qin@microsoft.com>
2024-12-20 22:57:04 +00:00
kuqin12
4b2f964749 BaseTools: Adding support of building BaseTool on Windows ARM/ARM64
This change focuses on the support of building basetool natively for
Windows ARM/ARM64 host system, which will enable the ARM based platforms
to build UEFI and unit tests.

Note that the warnings due to integer conversions are suppressed for
this specific target to avoid too much local changes carried in MU. The
formal change should drop all these binaries and move to pythonic
scripts.

The binary output is tested on Windows ARM based hardware systems.

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Kun Qin <kun.qin@microsoft.com>
2024-12-20 22:57:04 +00:00
Dionna Glaze
74ac8cc0e8 BaseTools: Typo fixes
Some in error messages, some in local variable names.

Signed-off-by: Dionna Glaze <dionnaglaze@google.com>
2024-12-11 09:00:21 +08:00
Rebecca Cran
5b2d55533b BaseTools: Improve error messages from UefiCapsuleHeader.py
Instead of throwing ValueErrors with no explanation, add a message
explaining what went wrong.

Signed-off-by: Rebecca Cran <rebecca@bsdio.com>
2024-12-06 13:02:18 +00:00
Ard Biesheuvel
051ef932bd BaseTools/GenFw X64: Detect GOTCPRELX relaxations applied by LLD
GenFw relies on static ELF relocation tables emitted by the linker (via
the --emit-relocs command line switch). These are different from the
dynamic relocations that a dynamic loader uses: static relocations are
emitted by the compiler/assembler, and consumed by the linker to
construct the executable. Only when the load address is a priori unknown
are dynamic relocations emitted, by the linker, in a format that the
dynamic loader can consume.

This distinction is relevant because only dynamic relocations cover the
GOT, and so GOT based indirections are better avoided. Unfortunately,
there are cases where the toolchain insists on emitting GOT based symbol
references, and so we have to deal with them in one of 2 ways:

- replace GOT based symbol references with direct references, so that
  the GOT entries themselves are no longer used, and can be ignored when
  generating the PE/COFF relocation tables (AARCH64 and RISCV64 take
  this approach);

- infer the locations of the GOT slots from the references appearing in
  the code, and emit PE/COFF relocations for them so that their contents
  will be fixed up appropriately.

The latter is the approach taken by GenFw for x86_64, which is the only
feasible approach for its ISA, given that GOT slots can be used as
memory operands in many different types of instructions, not all of
which can be converted straight-forwardly.

E.g.,

   movq   foo@GOTPCREL(%rip), %rax

can always be converted into

   leaq   foo(%rip), %rax

whereas

   cmpq   foo@GOTPCREL(%rip), %rax

can only be converted under the 32-bit position dependent code model,
into

   cmpq   $foo, %rax

and so the GOT references cannot be elided when generating position
independent code, which is what GenFw requires.

To remove the need for the linker to guess where the instructions start,
the ELF psABI for x86_64 specifies a couple of relaxable alternatives
for GOTPCREL, which are used to annotate particular classes of GOT
referencing instructions that may be relaxed to their non-GOT
counterparts.

There is no specification for what --emit-relocs is supposed to produce,
or whether or not its output is supposed to reflect such relaxations.
ld.bfd and LLD behave differently in this regard, and the latter may
emit R_X86_64_REX_GOTPCRELX relocations for MOV instructions that it
already has relaxed into LEA instructions. This means the displacement
in the instruction no longer refers to the GOT slot, but directly to the
object itself, and emitting a relocation is not only unnecessary, but
also harmful as the PE/COFF loader will corrupt the object when it
applies the relocations at startup.

Under the position independent code model, the only relaxation that the
linker could have applied for a R_X86_64_REX_GOTPCRELX relocation is MOV
to LEA, so detect whether the instruction is already LEA, and ignore the
relocation if that is the case.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
2024-10-23 09:52:54 +00:00
Mike Beaton
cc47e82703 BaseTools: Fix redefinition of UINT8_MAX in Decompress.c on XCODE5
This is part of a sequence of commits to restore build on the XCODE5
toolchain.

The definition is required on other toolchains, but on XCODE5 results
in a macro redefined error (from the existing value 255) from
/usr/include/stdint.h.

Signed-off-by: Mike Beaton <mjsbeaton@gmail.com>
2024-09-27 16:57:36 +00:00
Mike Beaton
6820004b3e BaseTools: Fix multiple 'invalid escape sequence' warnings in tests
In Python 3.12 invalid escape sequences in strings moved from
DeprecationWarning to SyntaxWarning
(ref https://docs.python.org/3/whatsnew/changelog.html#python-3-12-0-final
and search for gh-98401). In a future Python version this will become
SyntaxError.

Multiple instances of these SyntaxWarnings are currently printed when
running the BaseTools tests using Python 3.12 (though without actually
failing the affected tests).

This commit updates all lines which were causing this type of warning.

Typical examples which needed fixing are:

- "BaseTools\Source\Python" representing a path: "\S" and "\P" are invalid
escape sequences, therefore left unchanged, therefore the test works
(with a warning in Python 3.12). r"BaseTools\Source\Python" represents
the same string, but with escapes turned off completely thus no warning.

- Where '\t\s' is used as a regex pattern, then chr(9) + '\\s' is sent
to the regex parser (with a warning in Python 3.12) since '\s' is not a
valid Python escape sequence. This works correctly, though arguably for
the wrong reasons. r'\t\s' sends the same as '\\t\\s', as originally
intended and with no warning.

(Note that ' and " are not fundamentally different in Python.)

Signed-off-by: Mike Beaton <mjsbeaton@gmail.com>
2024-09-23 04:55:53 +00:00
Abdul Lateef Attar
222e2854fe BaseTools: Update RETURN_ERROR Macro in BaseTypes.h
This patch is to sync RETURN_ERROR macro with the
MdePkg/Include/Base.h

Ref: 1a89d9887f MdePkg:Update Return Error Macro in Base.h

Fixing RETURN_ERROR macro.
It is causing problem in Coverity Static analysis tool
as we are directly converting the UINT value to INTN.

Changing value from UINT to INTN might cause problema
Here we know that the values would not be in loss of data.
To increase the code quality and increase the static tool
analysis score we have to change it

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Abdul Lateef Attar <AbdulLateef.Attar@amd.com>
2024-09-20 09:02:41 +00:00
Taylor Beebe
cac0955658 BaseTools: Update Stack Cookie Logic
This patch updates the GenC logic to generate a random stack cookie value
for the stack check libraries. These random values improve security
for modules which cannot update the global intrinsics.

If the stack cookie value is randomized in the AutoGen.h file each
build, the build system will determine the module/library must be
rebuilt causing effectively a clean build every time. This also makes
binary reproducibility impossible.

This patch updates the early build scripts to create 32 and 64-bit JSON
files in the build output directory which each contain 100 randomized
stack cookie values for each bitwidth. If the JSON files are already
present, then they are not recreated which allows them to be stored and
moved to other builds for binary reproducibility. Because they are in
the build directory, a clean build will cause the values to be
regenerated.

The logic which creates AutoGen.h will read these JSON files and use a
hash of the module GUID (the hash seed is fixed in Basetools) to index
into the array of stack cookie values for the module bitwidth. This
model is necessary because there isn't thread-consistent data so we
cannot use a locking mechanism to ensure only one thread is writing to
the stack cookie files at a time. With this model, the build threads
only need to read from the files.

Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
2024-09-13 03:58:46 +00:00
Oliver Smith-Denny
61f9695f20 BaseTools: Remove Pip BaseTools
BaseTools was moved out to a separate repo and consumed as a pip
module by edk2 CI. This process has not led to the desired goals
of doing so, so this patch removes the pip based BaseTools from
edk2 CI.

The original goal of moving BaseTools to a pip module was
primarily to speed up the development process, as the old edk2
mailing list was slow. However, with edk2 moving to PRs, it now
actually slows the BaseTools development process to have to do
a PR in another repo, publish the module, and then make a PR
in edk2 to consume the new BaseTools. It also holds up using
the features in a new BaseTools in other PRs.

There were other goals of moving, such as allowing projects to
use the BaseTools outside of edk2. This can still be accomplished
outside of this PR, this PR simply stops edk2 CI from using the
pip module.

Continuous-integration-options: PatchCheck.ignore-multi-package

Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
2024-09-10 00:41:53 +00:00
Ken Lautner
50871ee0ec BaseTools: GenMake: FIx missing logs from GenMake.py
EdkLogger logs were not showing up as part of the build log output.
Adding the EdkLogger import to GenMake.py fixes the missing log prints.

Signed-off-by: Kenneth Lautner <kenlautner3@gmail.com>
2024-09-02 06:32:06 +00:00
Joey Vagedes
95ee7f3ef7 BaseTools: Trim: Add header/footer for ASL include
When including one ASL file in another, add a header / footer to the
included file to easily tell where the included file starts and ends.

Signed-off-by: Joey Vagedes <joey.vagedes@gmail.com>
2024-08-31 04:58:54 +00:00
Jason1 Lin
68b4c4b481 BaseTools/Capsule: Support Different Hash Algorithm for Payload Digest
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=4821

- The capsule payload digest got hardcoded inside the GenerateCapsule
  script as "sha256".

- It would be hard for the caller to change the supported hash algorithm
  which supported on OpenSSL or Windows signtool program and platform.

- Capsule payload digest signed data is followed by the PKCS#7 standard,
  in EDK-II CryptoPkg "Pkcs7Verify ()" is supported to validate with
  several hash algorithms naturally.
  (md5, sha1, sha256, sha384, and sha512)

- Deliver below changes within this patch,
  (1) Introduce an optional argument "--hash-algorithm" to assign
      the caller expected one and leave the default value "sha256"
      to support the backward compatibility.

  (2) Add the double quotes to put the string of certificate's
      subject name inside it.

  (3) Set "Open" argument of "SignToolSubjectName" into "False".

  (4) Set "Convert" argument of "SignToolSubjectName: into "str".

  (5) Correct the actual name of the "--subject-name" flag.

  (6) Add back correct number of arguments for PayloadDescriptor
      class object initializing.

Note:
- Platform needs to support the correspond hash algorithm to validate
  the digital signature or the failure would be observed.

- Set the md5 and sha1 algorithm as EOL based on the CryptoPkg supported
  table and reject the capsule creation.

Signed-off-by: Jason1 Lin <jason1.lin@intel.com>
2024-08-05 05:49:16 +00:00
Dmitry Antipov
41426040da BaseTools: Move MEMORY_TYPE_* Defines to EFI_MEMORY_TYPE Enum
Per TCBZ2372, clang on Linux emits a warning if an enum-typed variable
is compared with a constant outside of the range of the enum. Such
comparisons are performed in multiple locations in DXE core on
variables of type EFI_MEMORY_TYPE. This patch moves the OEM and OS
reserved types into the EFI_MEMORY_TYPE enum itself to resolve this
issue and improve readability. This commit does this for the BaseTools
copy of this enum.

Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
2024-07-26 03:08:20 +00:00
Leif Lindholm
5366def8d0 BaseTools: drop GeneralCheckNonAscii() from ECC
The GeneralCheckNonAscii() function is a sledgehammer rejecting any file
containing any character outside of the 7-bit ASCII encoding space, as
well as the DEL character (which seems unrelated).

This conflicts with basic stuff like correctly spelling certain proper
nouns in comments (like copyright statements), or string literals (for
example in multi-language driver binding ComponentNames).

So rip it out, to be replaced by more fine-grained checks to be added as
identified and needed.

Signed-off-by: Leif Lindholm <quic_llindhol@quicinc.com>
2024-07-15 13:27:00 +00:00
Leif Lindholm
8ade6d7bd1 BaseTools: fix consistent Ecc misspelling of ASCII
Ecc concistently referred to ASCII/Ascii as ACSII/Acsii, which
bugged me to no end when trying to figure out how those tests
worked. Fix all instances.

Signed-off-by: Leif Lindholm <quic_llindhol@quicinc.com>
2024-07-15 13:27:00 +00:00
Michael Kubacki
6c061c4715 BaseTools/Ecc: Allow static as a modifier
Currently, `STATIC` is allowed as a function modifier but `static`
results in the below ECC errors:

```
*Error code: 5001
*Return type of a function should exist and in the first line
*file: D:\src\edk2\Build\.pytool\Plugin\EccCheck\MdePkg\Library\UefiDebugLibDebugPortProtocol\DebugLibConstructor.c
*Line number: 37
*[UefiDebugLibDebugPortProtocolExitBootServicesCallback] Return
 Type should appear at the start of line

EFI coding style error
*Error code: 5002
*Any optional functional modifiers should exist and next to the
 return type
*file: D:\src\edk2\Build\.pytool\Plugin\EccCheck\MdePkg\Library\UefiDebugLibDebugPortProtocol\DebugLibConstructor.c
*Line number: 37
```

This is because `GetDataTypeFromModifier()` will return both `static`
and the return type (e.g. `VOID`) whereas for a modifier in the list
(e.g. `STATIC`) it will return only the return type allowing logic in
Ecc/c.py to process the modifier and return type with current logic.

Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
2024-07-10 14:32:08 +00:00
Sergii Dmytruk
26bc42f1e3 BaseTools/GenerateCapsule.py: Fix checking for DepExp presence
struct.unpack() returns a tuple even for a single-element pack,
resulting in signature verification being evaluated to false even when
the signature is there.

This fixes --decode and --dump-info actions incorrectly reporting issues
with parsing capsule dependencies when there are none.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Sergii Dmytruk
eeddb86aaa BaseTools/GenerateCapsule.py: Fix inconsistent error formatting
Just add a space between colon and a more detailed error message in two
places.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Sergii Dmytruk
47c1078175 BaseTools/GenerateCapsule.py: Require --output for --decode
--decode unconditionally uses args.OutputFile.name as a prefix for
output files that it creates and fails in a non-pretty way without
--output option.

This doesn't address creation/truncation of the file specified via
--output, but at least you're able to decode a capsule.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Sergii Dmytruk
822ff966c6 BaseTools/GenerateCapsule.py: Better error message on --decode failure
Print error text from the exception.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Sergii Dmytruk
3be79ece37 BaseTools/GenerateCapsule.py: Disallow UpdateImageIndex == 0 on --encode
This field seems to be one-based according UEFI specification, default
value is 1 and --decode of GenerateCapsule.py errors upon seeing
UpdateImageIndex less than 1.  So align --encode behaviour to enforce a
value within the 1..255 range.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Sergii Dmytruk
8e7bd66dc1 BaseTools/GenerateCapsule.py: Fix --decode operation
Commit b68d566439 added support for
input subject name with signtool and broke --decode operation by using
incorrect identifier in one place (could be an incomplete rename during
review).

It's `args.SignToolSubjectName`, not `args.SignSubjectName`.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Sergii Dmytruk
5a4a7172bc BaseTools/FmpCapsuleHeader.py: Explain error when throwing exceptions
This gives a caller a chance to report a meaningful error to the user.

Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
2024-07-05 19:33:50 +00:00
Joey Vagedes
6b9307192b BaseTools: InfBuildData: Fix Private value retrieval
Update retrieval of private guids, protocols, or ppis from a package's
declaration file to use the original path of the module's INF file
rather than the current path. When building the same module multiple
times in the same INF (by override the define's FILE_GUID), a temporary
instance of the module is generated outside the package, causing the
retrieval of private values to fail as the check to access private
values is done by verifying the module to build, is inside the package.

Signed-off-by: Joey Vagedes <Joey.Vagedes@gmail.com>

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
2024-07-04 11:55:25 +00:00
Jeff Brasen
aa99d36be9 BaseTools/BuildReport: Improve compile_commands generation
This produces output that matches CodeChecker log command

- Set directory to build output path
- Set build destination to the object created instead of the path
- Add recursive macro support
- Add lookup in module.Macros dictionary
- Add leading include flag to include list
- Add source file to compile commands

Signed-off-by: Jeff Brasen <jbrasen@nvidia.com>
2024-06-15 11:07:28 +00:00
Gua Guo
d97f964f7c BaseTools/Fmmt.py: Python 3.12 support
Ref to https://docs.python.org/3/whatsnew/3.12.html
A backslash-character pair that is not a valid
escape sequence now generates

Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
Reviewed-by: Liming Gao <gaoliming@byosoft.com.cn>
2024-04-23 21:36:22 +00:00