Running the vulture tool on the UPT/Xml folder gave the following
report. Remove the unnecessary code.
- UPT/Xml/CommonXml.py:585:
unused attribute 'LangDefsList' (60% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Running the vulture tool on the UPT/PomAdapter folder gave the following
report. Remove the unnecessary code.
- UPT/PomAdapter/DecPomAlignment.py:898:
unused method 'ShowPackage' (60% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Running the vulture tool on the Capsule folder gave the following
report. Remove the unnecessary code.
- Capsule/WindowsCapsuleSupportHelper.py:26:
unused method 'RegisterHelpers' (60% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Running the vulture tool gave the following report.
Remove the unreachable code.
- TargetTool/TargetTool.py:49:
unreachable code after 'raise' (100% confidence)
- UPT/Library/UniClassObject.py:137:
unreachable code after 'return' (100% confidence)
- UPT/Object/Parser/InfDefineObject.py:795:
unreachable code after 'if' (100% confidence)
- Ecc/Check.py:1504:
unreachable code after 'return' (100% confidence)
Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
Adds support for building the C language BaseTools for Windows using
toolchains based on mingw-w64.
Mingw-w64 is a collection of header files, libraries, and tools that
when combined with a compiler enable development of Windows software.
Mingw-w64 is a fork of the original MinGW (Minimalist GNU for Windows).
Most active development on MinGW has ceased and mingw-w64 is now the
actively maintained successor. Mingw-w64 provides a libc implementation
built on top of Microsoft's UCRT (Universal C Runtime) with all
nessesary compiler bindings needed to support the C++11 feature set.
Modern mingw-w64 development appears to have coalesced around MSYS2,
which produces a distributions of both GCC and LLVM/Clang that use
mingw-w64 to target the Windows OS. This MSYS2 Clang distribution has a
UNIX-like directory layout and includes Windows binaries of GNU Make.
Combined with the open source licensing, MSYS2's Clang distribution is a
highly attractive choice as an alternative Windows SDK for open source
projects such as TianoCore.
If one wishes to use EDK II to build UEFI firmware on the Windows
platform, then the C BaseTools need to be compiled as Windows
applications. This includes the PcdValueInit.exe program, which needs
to be recompiled every time a firmware build is run in order to
regenerate the initial values for structured PCDs. Currently, BaseTools
only supports the Visual C++ toolchain on the Windows platform. The
following new features have been added to enable usage of the toolchains
derived from mingw-w64:
- Fixes to the BaseTools C source code to support the use of a
GCC-style compiler on the Windows OS.
- The GNU Make-style Makefiles for the C BaseTools have been modified
to support Windows. Both GCC + mingw-w64 and Clang + mingw-w64 have
been tested and confirmed to build a working BaseTools.
- BaseTools now supports generating GNU Make-style Makefiles on the
Windows platform for the purpose of building firmware.
- edksetup.bat has been modified to optionally build BaseTools via
mingw-w64. There is no impact to the existing support for Visual C++
and Visual C++ remains the default toolchain.
Usage Instructions:
For the vast majority of users, the only system setup change nessesary
to use a mingw-w64 toolchain is to set the BASETOOLS_MINGW_PATH to the
directory containing the desired mingw-w64 based toolchain.
A new command line argument has been added to edksetup.bat: Mingw-w64
If this command line argument is set, then the script will set the
BASETOOLS_MINGW_BUILD environment variable. The user can also opt to set
this environment variable manually before running edksetup.bat
If BASETOOLS_MINGW_BUILD is defined, then the BASETOOLS_MINGW_PATH
environment variable must point to the directory containing the
mingw-w64 toolchain.
If CLANG_BIN is not defined and %BASETOOLS_MINGW_PATH%\bin\clang.exe
exists, then edksetup.bat will set CLANG_BIN=%BASETOOLS_MINGW_PATH%\bin\
This removes the requirement to configure the CLANG_BIN environment
variable manually in order to run a CLANGPDB or CLANGDWARF build if one
has the MSYS2 Clang distribution installed. If one wishes to use a
different copy of Clang (for example official LLVM binaries) to build
firmware and only use the MSYS2 Clang to build BaseTools, then one can
continue to set the CLANG_BIN environment variable, same as before. I
have tested the MSYS2 Clang distribution against the official LLVM
distribution and can confirm that if the compiler version is the same
the emitted machine code is identical between the two. Interestingly,
the MSYS2 Clang distribution emits the path to the PDB file using "/" as
the path seperator instead of "\". That appears to be the only
difference in output. Therefore, using the MSYS2 Clang distribution to
compile firmware seems a reasonable choice.
If CLANG_HOST_BIN is not defined and BASETOOLS_MINGW_BUILD is defined
and %BASETOOLS_MINGW_PATH%\bin\mingw32-make.exe exists, then
edksetup.bat will add %BASETOOLS_MINGW_PATH%\bin\ to the PATH and set
CLANG_HOST_BIN=mingw32-
This enable usage of the GNU Make included in the mingw-w64 toolchain
to build firmware in addition to BaseTools. if BASETOOLS_MINGW_BUILD is
not defined, edksetup.bat will continue to set CLANG_HOST_BIN=n, which
uses nmake to build firmware. This behavior can be overridden by
manually setting the value of CLANG_HOST_BIN before executing
edksetup.bat if one wishes to use a specific Make utility for the
CLANGPDB/CLANGDWARF toolchains.
References:
- https://www.mingw-w64.org/
- https://www.msys2.org/
Co-authored-by: Sandesh Jain <sandesh.jain@intel.com>
Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
Add GUID section for build report.
Also, change the GUID format to string format to be easier to parse
Signed-off-by: Zhiguang Liu <zhiguang.liu@intel.com>
BaseTools currently does not expand macros for component architecture when
nested !include directives are used. This breaks use cases like
[Component.$(DXE_ARCH)].
The fix is to add explicit macro expansion when computing the Arch value
for component lines in DscParser.
Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
The following example fails to be parsed correctly due to Size
being used in the outer scope but initialized in the inner
scope
```
gPlatformPkgTokenSpaceGuid.PcdSecureBootDbxBinaryFile|{}
```
Problematic code:
```python
for Item in NewPcdValueList:
Size = 0
# ....
if Size > 0:
PcdValue = '{' + ', '.join(AllPcdValueList) + '}'
````
Signed-off-by: Doug Flick <dougflick@microsoft.com>
When using the replace function, if the replaced file
located in a compress section, then the tool will re-build
the section. As the compress status set in the wrong place,
in some situation, the compress will do twice times which is
a wrong behavior.
This patch is used to fix this issue.
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Ashraf Ali S <ashraf.ali.s@intel.com>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
When a module M depends on L1, which depends on L2,
which depends on L3, the build fails when the library instance
of L3 cannot be found according to the library class-instance
mapping configuration specified in the DSC file.
When such failure happens, the build tool only prints that the
instance of L3 required by module M cannot be found. But it
does not tell how L3 is required by M.
The change enhances build tool to print the entire dependency
chain when such failure happens.
With the change, the new error message will be as follows:
<dsc-path>(...): error 4000: Instance of library class [L3] is not
found for module [M], [L3] is:
consumed by <instance of L2>
consumed by <instance of L1>
Signed-off-by: Ray Ni <ray.ni@intel.com>
If an FV_SPARE_SPACE_THRESHOLD is enabled and an FV
is 100% full with 0 bytes free, then this is likely
a special FV that may have alignment requirements
for the FFS file for both the start address and the
length and the FFS file consumes all the available
FV space.
Reduce FV_SPARE_SPACE_THRESHOLD from an error to a
warning if this FV 0 bytes free condition is
detected.
PR #10828 introduced the generation of these error
conditions for an FV with large alignment requirements.
The pad region before the aligned FFS file used to
be counted as free space even though it could never
be used due to the alignment requirements. There was
actually no free space available. PR #10828 fixed the
free space calculation to properly show it as 0 bytes
free, and this change then caused build error when
FV_SPARE_SPACE_THRESHOLD feature was enabled. The
reduction to a warning for this condition allows the
build to complete with errors and also provides a
build log warning message for review.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
- FMMT tool would use the "PATH" environment variable for locating
the required GUID tool.
- On Windows-like system, batch file not found in the "PATH" environment
variable when "shell=False".
- This issue required commands to include program extensions or
absolute paths.
- This patch sets "shell=True" to extend the support for batch files,
including scripts in BinWrappers under BaseTools.
- Converted input commands from lists to strings to ensure proper
argument interpretation in POSIX-like shell scripts.
Signed-off-by: Jason1 Lin <jason1.lin@intel.com>
If command line options are moved into a response file
of a GCC family build, then the file path separators are
converted from '\' to '/'. However, this can corrupt
command line options that are quoted strings.
Update GenMake to no convert '\' to '/' in quoted strings.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
This patch addresses an issue in the FMMT operations where the parent
reference is not checked for NoneType. This oversight can lead to an
AttributeError: 'NoneType' object has no attribute 'Name' when
attempting to access the parent reference. The fix involves adding a
check for NoneType before accessing the parent reference to ensure that
the operations handle such cases gracefully.
The affected functions include:
- AddNewFfs
- ReplaceFfs
- ExtractFfs
These functions now include proper checks to prevent the AttributeError.
Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
Refer to the docs of python, `os.path.normcase(path)` function:
"Normalize the case of a pathname. On Windows, convert all characters in
the pathname to lowercase, and also convert forward slashes to backward
slashes. On other operating systems, return the path unchanged."
`os.path.normpath(path)` also convert forward slashes to backward slashes.
So call `os.path.normcase` after `os.path.normpath` just convert path to
lowercase on Windows(only).
And Windows is case-insensitive but case-preserving.
So the usage of `os.path.normcase(os.path.normpath(path))` can be
simplified to `os.path.normpath(path)`. Then we can use case-preserving
paths rather than lowercase paths in compile_commands.json file
or build log.
But this patch continue to use `os.path.normcase`
when comparing/searching paths.
Signed-off-by: Yang Gang <yanggang@byosoft.com.cn>
In Python 3.12 invalid escape sequences in strings moved from
DeprecationWarning to SyntaxWarning
(ref https://docs.python.org/3/whatsnew/changelog.html#python-3-12-0-final
and search for gh-98401). In a future Python version this will become
SyntaxError.
Multiple instances of these SyntaxWarnings are currently printed when
running the BaseTools tests using Python 3.12 (though without actually
failing the affected tests).
This commit updates all lines which were causing this type of warning.
Typical examples which needed fixing are:
- "BaseTools\Source\Python" representing a path: "\S" and "\P" are invalid
escape sequences, therefore left unchanged, therefore the test works
(with a warning in Python 3.12). r"BaseTools\Source\Python" represents
the same string, but with escapes turned off completely thus no warning.
- Where '\t\s' is used as a regex pattern, then chr(9) + '\\s' is sent
to the regex parser (with a warning in Python 3.12) since '\s' is not a
valid Python escape sequence. This works correctly, though arguably for
the wrong reasons. r'\t\s' sends the same as '\\t\\s', as originally
intended and with no warning.
(Note that ' and " are not fundamentally different in Python.)
Signed-off-by: Mike Beaton <mjsbeaton@gmail.com>
This patch updates the GenC logic to generate a random stack cookie value
for the stack check libraries. These random values improve security
for modules which cannot update the global intrinsics.
If the stack cookie value is randomized in the AutoGen.h file each
build, the build system will determine the module/library must be
rebuilt causing effectively a clean build every time. This also makes
binary reproducibility impossible.
This patch updates the early build scripts to create 32 and 64-bit JSON
files in the build output directory which each contain 100 randomized
stack cookie values for each bitwidth. If the JSON files are already
present, then they are not recreated which allows them to be stored and
moved to other builds for binary reproducibility. Because they are in
the build directory, a clean build will cause the values to be
regenerated.
The logic which creates AutoGen.h will read these JSON files and use a
hash of the module GUID (the hash seed is fixed in Basetools) to index
into the array of stack cookie values for the module bitwidth. This
model is necessary because there isn't thread-consistent data so we
cannot use a locking mechanism to ensure only one thread is writing to
the stack cookie files at a time. With this model, the build threads
only need to read from the files.
Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
BaseTools was moved out to a separate repo and consumed as a pip
module by edk2 CI. This process has not led to the desired goals
of doing so, so this patch removes the pip based BaseTools from
edk2 CI.
The original goal of moving BaseTools to a pip module was
primarily to speed up the development process, as the old edk2
mailing list was slow. However, with edk2 moving to PRs, it now
actually slows the BaseTools development process to have to do
a PR in another repo, publish the module, and then make a PR
in edk2 to consume the new BaseTools. It also holds up using
the features in a new BaseTools in other PRs.
There were other goals of moving, such as allowing projects to
use the BaseTools outside of edk2. This can still be accomplished
outside of this PR, this PR simply stops edk2 CI from using the
pip module.
Continuous-integration-options: PatchCheck.ignore-multi-package
Signed-off-by: Oliver Smith-Denny <osde@linux.microsoft.com>
EdkLogger logs were not showing up as part of the build log output.
Adding the EdkLogger import to GenMake.py fixes the missing log prints.
Signed-off-by: Kenneth Lautner <kenlautner3@gmail.com>
When including one ASL file in another, add a header / footer to the
included file to easily tell where the included file starts and ends.
Signed-off-by: Joey Vagedes <joey.vagedes@gmail.com>
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=4821
- The capsule payload digest got hardcoded inside the GenerateCapsule
script as "sha256".
- It would be hard for the caller to change the supported hash algorithm
which supported on OpenSSL or Windows signtool program and platform.
- Capsule payload digest signed data is followed by the PKCS#7 standard,
in EDK-II CryptoPkg "Pkcs7Verify ()" is supported to validate with
several hash algorithms naturally.
(md5, sha1, sha256, sha384, and sha512)
- Deliver below changes within this patch,
(1) Introduce an optional argument "--hash-algorithm" to assign
the caller expected one and leave the default value "sha256"
to support the backward compatibility.
(2) Add the double quotes to put the string of certificate's
subject name inside it.
(3) Set "Open" argument of "SignToolSubjectName" into "False".
(4) Set "Convert" argument of "SignToolSubjectName: into "str".
(5) Correct the actual name of the "--subject-name" flag.
(6) Add back correct number of arguments for PayloadDescriptor
class object initializing.
Note:
- Platform needs to support the correspond hash algorithm to validate
the digital signature or the failure would be observed.
- Set the md5 and sha1 algorithm as EOL based on the CryptoPkg supported
table and reject the capsule creation.
Signed-off-by: Jason1 Lin <jason1.lin@intel.com>
The GeneralCheckNonAscii() function is a sledgehammer rejecting any file
containing any character outside of the 7-bit ASCII encoding space, as
well as the DEL character (which seems unrelated).
This conflicts with basic stuff like correctly spelling certain proper
nouns in comments (like copyright statements), or string literals (for
example in multi-language driver binding ComponentNames).
So rip it out, to be replaced by more fine-grained checks to be added as
identified and needed.
Signed-off-by: Leif Lindholm <quic_llindhol@quicinc.com>
Ecc concistently referred to ASCII/Ascii as ACSII/Acsii, which
bugged me to no end when trying to figure out how those tests
worked. Fix all instances.
Signed-off-by: Leif Lindholm <quic_llindhol@quicinc.com>
Currently, `STATIC` is allowed as a function modifier but `static`
results in the below ECC errors:
```
*Error code: 5001
*Return type of a function should exist and in the first line
*file: D:\src\edk2\Build\.pytool\Plugin\EccCheck\MdePkg\Library\UefiDebugLibDebugPortProtocol\DebugLibConstructor.c
*Line number: 37
*[UefiDebugLibDebugPortProtocolExitBootServicesCallback] Return
Type should appear at the start of line
EFI coding style error
*Error code: 5002
*Any optional functional modifiers should exist and next to the
return type
*file: D:\src\edk2\Build\.pytool\Plugin\EccCheck\MdePkg\Library\UefiDebugLibDebugPortProtocol\DebugLibConstructor.c
*Line number: 37
```
This is because `GetDataTypeFromModifier()` will return both `static`
and the return type (e.g. `VOID`) whereas for a modifier in the list
(e.g. `STATIC`) it will return only the return type allowing logic in
Ecc/c.py to process the modifier and return type with current logic.
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
struct.unpack() returns a tuple even for a single-element pack,
resulting in signature verification being evaluated to false even when
the signature is there.
This fixes --decode and --dump-info actions incorrectly reporting issues
with parsing capsule dependencies when there are none.
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
--decode unconditionally uses args.OutputFile.name as a prefix for
output files that it creates and fails in a non-pretty way without
--output option.
This doesn't address creation/truncation of the file specified via
--output, but at least you're able to decode a capsule.
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
This field seems to be one-based according UEFI specification, default
value is 1 and --decode of GenerateCapsule.py errors upon seeing
UpdateImageIndex less than 1. So align --encode behaviour to enforce a
value within the 1..255 range.
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
Commit b68d566439 added support for
input subject name with signtool and broke --decode operation by using
incorrect identifier in one place (could be an incomplete rename during
review).
It's `args.SignToolSubjectName`, not `args.SignSubjectName`.
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
Update retrieval of private guids, protocols, or ppis from a package's
declaration file to use the original path of the module's INF file
rather than the current path. When building the same module multiple
times in the same INF (by override the define's FILE_GUID), a temporary
instance of the module is generated outside the package, causing the
retrieval of private values to fail as the check to access private
values is done by verifying the module to build, is inside the package.
Signed-off-by: Joey Vagedes <Joey.Vagedes@gmail.com>
Cc: Rebecca Cran <rebecca@bsdio.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Yuwei Chen <yuwei.chen@intel.com>
This produces output that matches CodeChecker log command
- Set directory to build output path
- Set build destination to the object created instead of the path
- Add recursive macro support
- Add lookup in module.Macros dictionary
- Add leading include flag to include list
- Add source file to compile commands
Signed-off-by: Jeff Brasen <jbrasen@nvidia.com>
