build_make, branch replicant-11 Fork of build/make Include vendor/replicant/build's core/main.mk 2021-12-10T15:29:15+00:00 Denis 'GNUtoo' Carikli GNUtoo@cyberdimension.org 2021-12-10T15:24:02+00:00 bec1429941b88347f341e8da08f27bfbf731301d To use ADDITIONAL_BUILD_PROPERTIES in vendor/replicant/build, we need to include its core/main.mk here otherwise we have the following build error: build/make/core/main.mk:127: error: ADDITIONAL_BUILD_PROPERTIES must not be set before here: ro.replicant.build.version=11.0.alpha. ckati failed with: exit status 1 This commit is based on the followin commit from LineageOS's build/make repository[1]: 261741115e9a2e5a933810613dc7a3bfb5612f00 (build: Add Lineage build support) [1]The commit is available in the lineage-18.1 branch of https://github.com/LineageOS/android_build We removed the leading '-' used in the LineageOS commit as we want the build to fail if for some reason vendor/replicant/build/core/main.mk is not there. Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> 
To use ADDITIONAL_BUILD_PROPERTIES in vendor/replicant/build, we need
to include its core/main.mk here otherwise we have the following build
error:
    build/make/core/main.mk:127: error: ADDITIONAL_BUILD_PROPERTIES
    must not be set before here:
    ro.replicant.build.version=11.0.alpha.
    ckati failed with: exit status 1

This commit is based on the followin commit from LineageOS's
build/make repository[1]:
    261741115e9a2e5a933810613dc7a3bfb5612f00
    (build: Add Lineage build support)

[1]The commit is available in the lineage-18.1 branch of
   https://github.com/LineageOS/android_build

We removed the leading '-' used in the LineageOS commit as we want the
build to fail if for some reason vendor/replicant/build/core/main.mk
is not there.

Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
Build separate recovery kernel image 2021-09-27T15:13:38+00:00 Denis 'GNUtoo' Carikli GNUtoo@cyberdimension.org 2021-09-27T15:13:13+00:00 41141d9d39d3aaeea6e97f21608f0323ce792494 Even if we use the highest compression ratio support for the recovery and the kernel image as on the Galaxy SIII (GT-I9300) and Galaxy SIII 4G (GT-I9305), the recovery partition is 8MiB and without removing options that are not used in the recovery (like the WiFi, netfilter, etc) we still don't fit in the 8MiB. With a specific defconfig we however can manage to make it fit but we don't have a lot of margin as we get the following warning while building Replicant images: WARNING: +out/target/product/i9300/recovery.img approaching size limit (8370176 now; limit 8388608) Having the smallest kenrel image and initramfs image in the recovery bootimage has been chosen here as we cannot simply fit a recovery initramfs in the RECOVERY partition without any kernel because even if Android 11 supports that, if have a bootimage without any kernel or bootloader code instead, rebooting to recovery or booting to recovery does result in the device booting directly to the dowload mode and we have a screen with the following: [Android logo] Downloading.... Do not rurn off target !! and then there is no other way to make the device boot to the BOOT partition other than flashing code (like a recovery) to the RECOVERY partition in order to make it reboot to the normal BOOT partition. And here, even 'heimdall close-pc-screen' doesn't change the boot mode. While writing some code code reboot the device to the BOOT partition is trivial to do[1], the result is not totally transparent as pressing the power up, menu and power button don't result in the device booting to the RECOVERY. In addition it would probably complexify a lot the installation instructions because, when the recovery initramfs is on a separate partition, the kenrel still depends on the init daemon that is inside the SYSTEM partition to be able to boot the recovery. And Installing big images (like system.img) with heimdall is not very reliable on the Exynos devices that are supported by Replicant. So we are left with two possibilities: we can either: - Write a tiny bootloader that would be installed in the RECOVERY bootimage instead of the kernel, and that would retrieve the kernel from the BOOT bootimage, and boot the kernel with the recovery initramfs. - Shrink as much as possible the kernel image used for the recovery and the recovery initramfs. Doing the former would require to write code to retrieve data from eMMC partitions (and probably support all eMMC standards as we don't have a list of possible eMMCs and that some could come from reparations having been made), parse the bootimages, and write code to retrieve the boot arguments, because we can't tell the kernel to load a recovery image from a partition as the kernel doesn't have support for that yet. As we also need a bootimage in the RECOVERY partition, we would also probably need tricks to add a compressed squashfs that would be treated as block device. If we manage to do that we would still need to modify a bit the installation instructions, and this would break the many unofficial tutorials (often in local languages) that are not made by Replicant. As for the later it might increase too much the maintenance work if the kernel keeps growing and that we spend lot of time tuning the defconfig to make it fit, but for now shrinking the kernel and initramfs works, so we could write or adapt a bootloader later if the maintenance of the recovery defconfig turns out to takes too much time. [1]https://git.replicant.us/contrib/GNUtoo/tools/s-boot_usb_exploit/tree/shellcode/reboot.c Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> 
Even if we use the highest compression ratio support for the
recovery and the kernel image as on the Galaxy SIII (GT-I9300)
and Galaxy SIII 4G (GT-I9305), the recovery partition is 8MiB
and without removing options that are not used in the recovery
(like the WiFi, netfilter, etc) we still don't fit in the 8MiB.

With a specific defconfig we however can manage to make it fit
but we don't have a lot of margin as we get the following
warning while building Replicant images:
    WARNING: +out/target/product/i9300/recovery.img
    approaching size limit (8370176 now; limit 8388608)

Having the smallest kenrel image and initramfs image in
the recovery bootimage has been chosen here as we cannot
simply fit a recovery initramfs in the RECOVERY
partition without any kernel because even if Android 11
supports that, if have a bootimage without any kernel or
bootloader code instead, rebooting to recovery or booting
to recovery does result in the device booting directly to
the dowload mode and we have a screen with the following:
    [Android logo]
    Downloading....
    Do not rurn off target !!
and then there is no other way to make the device boot to
the BOOT partition other than flashing code (like a
recovery) to the RECOVERY partition in order to make it
reboot to the normal BOOT partition.

And here, even 'heimdall close-pc-screen' doesn't change
the boot mode.

While writing some code code reboot the device to the BOOT
partition is trivial to do[1], the result is not totally
transparent as pressing the power up, menu and power button
don't result in the device booting to the RECOVERY.

In addition it would probably complexify a lot the
installation instructions because, when the recovery
initramfs is on a separate partition, the kenrel still
depends on the init daemon that is inside the SYSTEM
partition to be able to boot the recovery.

And Installing big images (like system.img) with heimdall
is not very reliable on the Exynos devices that are
supported by Replicant.

So we are left with two possibilities: we can either:
- Write a tiny bootloader that would be installed in
  the RECOVERY bootimage instead of the kernel, and that
  would retrieve the kernel from the BOOT bootimage, and
  boot the kernel with the recovery initramfs.
- Shrink as much as possible the kernel image used for the
  recovery and the recovery initramfs.

Doing the former would require to write code to retrieve
data from eMMC partitions (and probably support all eMMC
standards as we don't have a list of possible eMMCs and
that some could come from reparations having been made),
parse the bootimages, and write code to retrieve the boot
arguments, because we can't tell the kernel to load a
recovery image from a partition as the kernel doesn't
have support for that yet. As we also need a bootimage
in the RECOVERY partition, we would also probably need
tricks to add a compressed squashfs that would be treated
as block device.

If we manage to do that we would still need to modify a
bit the installation instructions, and this would break
the many unofficial tutorials (often in local languages)
that are not made by Replicant.

As for the later it might increase too much the maintenance
work if the kernel keeps growing and that we spend lot of
time tuning the defconfig to make it fit, but for now
shrinking the kernel and initramfs works, so we could write
or adapt a bootloader later if the maintenance of the
recovery defconfig turns out to takes too much time.

[1]https://git.replicant.us/contrib/GNUtoo/tools/s-boot_usb_exploit/tree/shellcode/reboot.c

Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
Add support for XZ recoveries 2021-09-25T16:57:28+00:00 Denis 'GNUtoo' Carikli GNUtoo@cyberdimension.org 2021-09-25T16:13:35+00:00 6b520e17e5ee4f4663e7fa452af7a6808e3858df We do need the highest compression ratio support for the recovery as on the Galaxy SIII (GT-I9300) and Galaxy SIII 4G (GT-I9305) the recovery partition is 8MiB and we need to fit a bootimage with inside a kernel image and a recovery initramfs. We cannot simply fit a recovery initramfs in the RECOVERY partition without any kernel because even if Android 11 supports that, if have a bootimage without any kernel or bootloader code instead, rebooting to recovery or booting to recovery does result in the device booting directly to the dowload mode and we have a screen with the following: [Android logo] Downloading.... Do not rurn off target !! and then there is no other way to make the device boot to the BOOT partition other than flashing code (like a recovery) to the RECOVERY partition in order to make it reboot to the normal BOOT partition. And here, even 'heimdall close-pc-screen' doesn't change the boot mode. While writing some code code reboot the device to the BOOT partition is trivial to do[1], the result is not totally transparent as pressing the power up, menu and power button don't result in the device booting to the RECOVERY. In addition it would probably complexify a lot the installation instructions because, when the recovery initramfs is on a separate partition, the kenrel still depends on the init daemon that is inside the SYSTEM partition to be able to boot the recovery. And Installing big images (like system.img) with heimdall is not very reliable on the Exynos devices that are supported by Replicant. So we are left with two possibilities: we can either: - Write a tiny bootloader that would be installed in the RECOVERY bootimage instead of the kernel, and that would retrieve the kernel from the BOOT bootimage, and boot the kernel with the recovery initramfs. - Shrink as much as possible the kernel image used for the recovery and the recovery initramfs. Doing the former would require to write code to retrieve data from eMMC partitions (and probably support all eMMC standards as we don't have a list of possible eMMCs and that some could come from reparations having been made), parse the bootimages, and write code to retrieve the boot arguments, because we can't tell the kernel to load a recovery image from a partition as the kernel doesn't have support for that yet. As we also need a bootimage in the RECOVERY partition, we would also probably need tricks to add a compressed squashfs that would be treated as block device. If we manage to do that we would still need to modify a bit the installation instructions, and this would break the many unofficial tutorials (often in local languages) that are not made by Replicant. So for now we did the later as for now we managed to shrink the kernel and recovery enough to work. To do that we used the kenrel's xz_wrap.sh script to ensure that the XZ options we use are compatible with Linux's xz decompressor. As XZ_WRAP variables are not used anywhere else (I've checked with repo grep XZ_WRAP) they are safe to use here. We do not set XZ as it's already set somewhere else and we probably don't want to use kernel specific XZ options for everything. [1]https://git.replicant.us/contrib/GNUtoo/tools/s-boot_usb_exploit/tree/shellcode/reboot.c Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> 
We do need the highest compression ratio support for the
recovery as on the Galaxy SIII (GT-I9300) and
Galaxy SIII 4G (GT-I9305) the recovery partition is 8MiB
and we need to fit a bootimage with inside a kernel image
and a recovery initramfs.

We cannot simply fit a recovery initramfs in the RECOVERY
partition without any kernel because even if Android 11
supports that, if have a bootimage without any kernel or
bootloader code instead, rebooting to recovery or booting
to recovery does result in the device booting directly to
the dowload mode and we have a screen with the following:
    [Android logo]
    Downloading....
    Do not rurn off target !!
and then there is no other way to make the device boot to
the BOOT partition other than flashing code (like a
recovery) to the RECOVERY partition in order to make it
reboot to the normal BOOT partition.

And here, even 'heimdall close-pc-screen' doesn't change
the boot mode.

While writing some code code reboot the device to the BOOT
partition is trivial to do[1], the result is not totally
transparent as pressing the power up, menu and power button
don't result in the device booting to the RECOVERY.

In addition it would probably complexify a lot the
installation instructions because, when the recovery
initramfs is on a separate partition, the kenrel still
depends on the init daemon that is inside the SYSTEM
partition to be able to boot the recovery.

And Installing big images (like system.img) with heimdall
is not very reliable on the Exynos devices that are
supported by Replicant.

So we are left with two possibilities: we can either:
- Write a tiny bootloader that would be installed in
  the RECOVERY bootimage instead of the kernel, and that
  would retrieve the kernel from the BOOT bootimage, and
  boot the kernel with the recovery initramfs.
- Shrink as much as possible the kernel image used for the
  recovery and the recovery initramfs.

Doing the former would require to write code to retrieve
data from eMMC partitions (and probably support all eMMC
standards as we don't have a list of possible eMMCs and
that some could come from reparations having been made),
parse the bootimages, and write code to retrieve the boot
arguments, because we can't tell the kernel to load a
recovery image from a partition as the kernel doesn't
have support for that yet. As we also need a bootimage
in the RECOVERY partition, we would also probably need
tricks to add a compressed squashfs that would be treated
as block device.

If we manage to do that we would still need to modify a
bit the installation instructions, and this would break
the many unofficial tutorials (often in local languages)
that are not made by Replicant.

So for now we did the later as for now we managed to shrink
the kernel and recovery enough to work.

To do that we used the kenrel's xz_wrap.sh script to ensure
that the XZ options we use are compatible with Linux's xz
decompressor.

As XZ_WRAP variables are not used anywhere else (I've checked with
repo grep XZ_WRAP) they are safe to use here.

We do not set XZ as it's already set somewhere else and we probably
don't want to use kernel specific XZ options for everything.

[1]https://git.replicant.us/contrib/GNUtoo/tools/s-boot_usb_exploit/tree/shellcode/reboot.c

Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
Fix the generation of installable zips 2021-09-20T16:27:46+00:00 Denis 'GNUtoo' Carikli GNUtoo@cyberdimension.org 2021-09-20T16:17:51+00:00 6681e4bb3dbbc6fbad5663055aff79483cb21b87 It's a good idea not to not change too much the way to install Replicant as otherwise: - The installation instructions would need to be modified in depth or rewritten completely for Replicant 11. - Users that already know how to install Replicant will have to learn new ways to do the installation if the previous was isn't available anymore. - Some non-official Replicant installation guides (typically in local languages) also exist. In addition it's really not a good idea to rely on heimdall to flash images that are too big as on the Exynos devices that we currently support in Replicant 6, you often need to retry several times to talk to the bootloader, and under high load on the computer running heimdall, it can fail in the middle of the flashing of an image. Howeve if we want to generate a zip file with ehtner 'make updatepackage' and 'make otapackage' for the i9300 target we have an error like that: 2021-09-20 18:15:46 - add_img_to_target_files.py - INFO : done. [100% 20/20] Package: out/target/product/i9300/replicant_i9300-img-eng.replicant.zip FAILED: out/target/product/i9300/replicant_i9300-img-eng.replicant.zip /bin/bash -c \ "PATH=[...] out/host/linux-x86/bin/img_from_target_files \ --additional \ IMAGES/VerifiedBootParams.textproto:VerifiedBootParams.textproto \ out/target/product/i9300/obj/PACKAGING/target_files_intermediates/replicant_i9300-target_files-eng.replicant.zip \ out/target/product/i9300/replicant_i9300-img-eng.replicant.zip" Traceback (most recent call last): File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 174, in _run_module_as_main File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 72, in _run_code File "[...]/out/host/linux-x86/bin/img_from_target_files/__main__.py", line 12, in <module> File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 174, in _run_module_as_main File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 72, in _run_code File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 246, in <module> File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 238, in main File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 191, in ImgFromTargetFiles File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 75, in LoadOptions File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 688, in LoadInfoDict File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 940, in _FindAndLoadRecoveryFstab File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 859, in LoadRecoveryFSTab File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 626, in read_helper File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 578, in ReadFromInputFile UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 30: ordinal not in range(128) 18:16:31 ninja failed with: exit status 1 After some time spent debugging the issue with print(), it failed with the RECOVERY/RAMDISK/system/etc/recovery.fstab file in the replicant_i9300-target_files-eng.replicant/ directory in the out/target/product/i9300/obj/PACKAGING/target_files_intermediates/ directory with both UTF-8 and POSIX locales set (it was verified with locale -a). Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> 
It's a good idea not to not change too much the way to install
Replicant as otherwise:
- The installation instructions would need to be modified in
  depth or rewritten completely for Replicant 11.
- Users that already know how to install Replicant will have
  to learn new ways to do the installation if the previous
  was isn't available anymore.
- Some non-official Replicant installation guides (typically
  in local languages) also exist.

In addition it's really not a good idea to rely on heimdall
to flash images that are too big as on the Exynos devices that
we currently support in Replicant 6, you often need to retry
several times to talk to the bootloader, and under high load
on the computer running heimdall, it can fail in the middle
of the flashing of an image.

Howeve if we want to generate a zip file with ehtner 'make updatepackage'
and 'make otapackage' for the i9300 target we have an error like that:
    2021-09-20 18:15:46 - add_img_to_target_files.py - INFO    : done.
    [100% 20/20] Package: out/target/product/i9300/replicant_i9300-img-eng.replicant.zip
    FAILED: out/target/product/i9300/replicant_i9300-img-eng.replicant.zip
    /bin/bash -c \
    "PATH=[...] out/host/linux-x86/bin/img_from_target_files \
    --additional \
    IMAGES/VerifiedBootParams.textproto:VerifiedBootParams.textproto \
    out/target/product/i9300/obj/PACKAGING/target_files_intermediates/replicant_i9300-target_files-eng.replicant.zip \
    out/target/product/i9300/replicant_i9300-img-eng.replicant.zip"
    Traceback (most recent call last):
      File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 174, in _run_module_as_main
      File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 72, in _run_code
      File "[...]/out/host/linux-x86/bin/img_from_target_files/__main__.py", line 12, in <module>
      File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 174, in _run_module_as_main
      File "[...]/out/host/linux-x86/bin/img_from_target_files/internal/stdlib/runpy.py", line 72, in _run_code
      File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 246, in <module>
      File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 238, in main
      File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 191, in ImgFromTargetFiles
      File "[...]/out/host/linux-x86/bin/img_from_target_files/img_from_target_files.py", line 75, in LoadOptions
      File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 688, in LoadInfoDict
      File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 940, in _FindAndLoadRecoveryFstab
      File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 859, in LoadRecoveryFSTab
      File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 626, in read_helper
      File "[...]/out/host/linux-x86/bin/img_from_target_files/common.py", line 578, in ReadFromInputFile
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 30: ordinal not in range(128)
    18:16:31 ninja failed with: exit status 1

After some time spent debugging the issue with print(), it failed
with the RECOVERY/RAMDISK/system/etc/recovery.fstab file in the
replicant_i9300-target_files-eng.replicant/ directory in the
out/target/product/i9300/obj/PACKAGING/target_files_intermediates/
directory with both UTF-8 and POSIX locales set (it was
verified with locale -a).

Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
Version bump to RP1A.201105.002 [core/build_id.mk] 2020-09-28T20:59:20+00:00 android-build-team Robot android-build-team-robot@google.com 2020-09-28T20:59:18+00:00 ea883a88bfe2e9bf0c2b532740bd433a910fb343 Change-Id: Ia242f24f086a1b16507923f96ae8eb76983172d9 
Change-Id: Ia242f24f086a1b16507923f96ae8eb76983172d9
Make change and version bump to RP1A.201105.001 2020-09-19T10:18:11+00:00 android-build-team Robot android-build-team-robot@google.com 2020-09-19T10:15:27+00:00 fee71dbba5238f541c84f854d84b582b63f7a124 Change-Id: I44c3f7ed47a695fb61da949f8405545072fd7b7d 
Change-Id: I44c3f7ed47a695fb61da949f8405545072fd7b7d
Update Security String to 2020-11-05 2020-09-19T10:10:54+00:00 Paul Scovanner pscovanner@google.com 2020-09-18T20:32:35+00:00 6c03196e119e1cda2923b8ce397baad611a4df0f Bug: 168709737 Change-Id: I981139f21fb2ffcc9688aad02839a81ce6de6f5f (cherry picked from commit 00d4c092a577ae19f80f56e29368ba2e442b5819) 
Bug: 168709737
Change-Id: I981139f21fb2ffcc9688aad02839a81ce6de6f5f
(cherry picked from commit 00d4c092a577ae19f80f56e29368ba2e442b5819)
Make change and version bump to RP1A.201101.001 2020-09-16T23:30:47+00:00 android-build-team Robot android-build-team-robot@google.com 2020-09-16T23:30:17+00:00 1c6d0fb34bce6e6463057c8bb4e8a8a5d4897429 Change-Id: I82e3d6832e8cb7a851ce9a5d2561437bd41f91e7 
Change-Id: I82e3d6832e8cb7a851ce9a5d2561437bd41f91e7
Update Security String to 2020-11-01 2020-09-16T23:28:18+00:00 Paul Scovanner pscovanner@google.com 2020-09-16T19:16:44+00:00 8c704a59cffd7ec18e47ebec7f3c740b737cf1e5 Bug: 168709737 Change-Id: I96489891caa4f5248c90be2d8c16327e1991e3bb (cherry picked from commit 727715f10066f193ec9aa56c188c1856db7c3214) 
Bug: 168709737
Change-Id: I96489891caa4f5248c90be2d8c16327e1991e3bb
(cherry picked from commit 727715f10066f193ec9aa56c188c1856db7c3214)
Version bump to RP1A.201005.006 [core/build_id.mk] 2020-09-11T19:01:21+00:00 android-build-team Robot android-build-team-robot@google.com 2020-09-11T19:01:19+00:00 f3a4604ae7286ec7df077ea48279655913d4b47e Change-Id: I18acc8afe854f5b4f38e8e479d3c893f9aace419 
Change-Id: I18acc8afe854f5b4f38e8e479d3c893f9aace419