Re: [yocto] [meta-intel] "Crazy" Xorg memory usage after upgrading from Daisy to Fido

[Cheah, Vincent Beng Keat]

Hi Chris, 

Sorry for the lateness in reply, was too busy with other things. Good to hear 
that.

Xorg.conf is basically to force load particular intended driver (in this case 
X11 SNA Driver). This is something that that we do not get when we bit-bake for 
a yocto image; mainly because users from different companies/purpose may want 
to use customized their own xorg.conf based on their needs.  You can check for 
which X11 driver that got loaded into your Yocto system simply by checking that 
on the /var/log/Xorg.0.log

In our lab, in order to make sure that our drivers are loaded correctly, we 
normally, build i915 as a module and add a xorg.conf files (containing some of 
the stuff below). 
Generally, in most of the case, we will get this loaded correctly (without 
Xorg.conf) but there do are times when we do not get the desired driver loaded. 

Section "Device"
Identifier  "Card0"
Driver  "intel"
BusID   "0:2:0"
Screen  0
EndSection

... Vincent 


-Original Message-
From: Chris Tapp [mailto:opensou...@keylevel.com] 
Sent: Wednesday, November 25, 2015 7:44 AM
To: Cheah, Vincent Beng Keat 
Cc: Yocto Project ; Chang, Rebecca Swee Fun 
; Paul Eggleton 
; meta-in...@yoctoproject.org
Subject: Re: [yocto] [meta-intel] "Crazy" Xorg memory usage after upgrading 
from Daisy to Fido

Hi Vincent,

I may have made some progress. The undesirable memory usage within Xorg isn’t 
there if I create an xorg.conf file containing:

Section “Device”
  Identifier “Intel Video”
  Driver “intel”
  Option “TearFree” “true"
EndSection

So it looks as if I need to enable “TearFree”. I didn’t need to add this for 
the 2.99.910 version of xf86-video-intel included with ‘daisy’.

Chris

> On 23 Nov 2015, at 23:48, Chris Tapp  wrote:
> 
> Hi Vincent,
> 
> I’ve finally got back to being able to investigate this further.
> 
> I’ve now moved on to “jethro” and I’m seeing exactly the same behaviour. I’ve 
> tried with kernel versions 3.14.39, 3.19.5 and 4.1.8.
> 
>> On 10 Jun 2015, at 03:50, Cheah, Vincent Beng Keat 
>>  wrote:
>> 
>> Hi Chris,
>> 
>> I don’t have any idea with regard to the issue that you are getting below. 
>> All the work that we are doing here so far is on CHV (yocto-kernel-3.19.5 
>> standard/base branch).
>> 
>> From your statement below, it looks to me that you are upgrading meta-intel 
>> from Daisy to Fido branch which are using yocto-kernel-3.14 
>> (meta-intel/isg/valleyisland BSP). I'm not sure if you are able to reproduce 
>> this with yocto-kernel-3.19.5 (standard/base branch) from the meta-intel 
>> common directory. Also, comparing Daisy branch against Fido, it seems like 
>> there are lot of changes in the user-space stacks, which I'm not sure could 
>> cause the issue below.
>> 
>> 
>> Daisy 1.6.2
>>  Kernel 3.4, 3.10, 3.14 (Supportable common base)
>>  Xorg-server 1.15
>>  Wayland/Weston 1.4.0
>>  Xf86-video-intel 2.99.910
>>  Libdrm 2.4.52
>>  MESA 9.2.5
>>  Cairo 1.12.16
>>  libVA 1.3.1 (from meta-intel)
>>  Intel-VA-driver 1.3.2 (from meta-intel)
>>  GStreamer 1.2.3
>>  GStreamer-VAAPI 0.5.8 (from meta-intel)
>> 
>> 
>> Dizzy 1.7.1
>>  Kernel 3.10, 3.14, 3.17 (Supportable common base)
>>  Xorg-server 1.15.1
>>  Wayland/Weston 1.5.0
>>  Xf86-video-intel 2.99.912
>>  Libdrm 2.4.54
>>  MESA 10.1.3
>>  Cairo 1.12.16
>>  libVA 1.3.1 (from meta-intel)
>>  Intel-VA-driver 1.3.2 (from meta-intel)
>>  GStreamer 1.4.1
>>  GStreamer-VAAPI 0.5.8 (from meta-intel)
>> 
>> 
>> Fido 1.8
>>  Kernel 3.14, 3.19 (supportable comon base)
>>  Xorg-server 1.16.3
>>  Wayland/weston 1.6.0
>>  Xf86-video-intel 2.99.917
>>  Libdrm 2.4.59
>>  Mesa 10.4.4
>>  Cairo 1.12.18
>>  LibVA 1.5.0 (from meta-intel)
>>  Intel-VA-driver 1.5.0 (from meta-intel)
>>  Gstreamer 1.4.5
>>  Gstreamer-vaapi 0.5.10 (from meta-intel)
>> 
>> 
>> ... Vincent
>> 
>> -Original Message-
>> From: Chang, Rebecca Swee Fun
>> Sent: Wednesday, June 10, 2015 9:08 AM
>> To: Cheah, Vincent Beng Keat
>> Cc: meta-in...@yoctoproject.org; Chris Tapp; Yocto Project; Wold, Saul; 
>> 'Paul Eggleton'
>> Subject: RE: [meta-intel] "Crazy" Xorg memory usage after upgrading 
>> from Daisy to Fido
>> 
>> Hi Vincent,
>> 
>> Can you help to comment on this issue mentioned by Chris?
>> Thanks.
>> 
>> Regards,
>> Rebecca
>> 
>>> -Original Message-
>>> From: Paul Eggleton [mailto:paul.eggle...@linux.intel.com]
>>> Sent: 09 June, 2015 12:15 AM
>>> To: Chang, Rebecca Swee Fun
>>> Cc: meta-in...@yoctoproject.org; Chris Tapp; Yocto Project; Wold, 
>>> Saul
>>> Subject: Re: [meta-intel] "Crazy" Xorg memory usage after upgrading 
>>> from Daisy to Fido
>>> 
>>> Rebecca, is this something you 

Re: [yocto] [meta-intel] "Crazy" Xorg memory usage after upgrading from Daisy to Fido

[Chris Tapp]

Hi Vincent,

I may have made some progress. The undesirable memory usage within Xorg isn’t 
there if I create an xorg.conf file containing:

Section “Device”
  Identifier “Intel Video”
  Driver “intel”
  Option “TearFree” “true"
EndSection

So it looks as if I need to enable “TearFree”. I didn’t need to add this for 
the 2.99.910 version of xf86-video-intel included with ‘daisy’.

Chris

> On 23 Nov 2015, at 23:48, Chris Tapp  wrote:
> 
> Hi Vincent,
> 
> I’ve finally got back to being able to investigate this further.
> 
> I’ve now moved on to “jethro” and I’m seeing exactly the same behaviour. I’ve 
> tried with kernel versions 3.14.39, 3.19.5 and 4.1.8.
> 
>> On 10 Jun 2015, at 03:50, Cheah, Vincent Beng Keat 
>>  wrote:
>> 
>> Hi Chris,
>> 
>> I don’t have any idea with regard to the issue that you are getting below. 
>> All the work that we are doing here so far is on CHV (yocto-kernel-3.19.5 
>> standard/base branch).
>> 
>> From your statement below, it looks to me that you are upgrading meta-intel 
>> from Daisy to Fido branch which are using yocto-kernel-3.14 
>> (meta-intel/isg/valleyisland BSP). I'm not sure if you are able to reproduce 
>> this with yocto-kernel-3.19.5 (standard/base branch) from the meta-intel 
>> common directory. Also, comparing Daisy branch against Fido, it seems like 
>> there are lot of changes in the user-space stacks, which I'm not sure could 
>> cause the issue below.
>> 
>> 
>> Daisy 1.6.2
>>  Kernel 3.4, 3.10, 3.14 (Supportable common base)
>>  Xorg-server 1.15
>>  Wayland/Weston 1.4.0
>>  Xf86-video-intel 2.99.910
>>  Libdrm 2.4.52
>>  MESA 9.2.5
>>  Cairo 1.12.16
>>  libVA 1.3.1 (from meta-intel)
>>  Intel-VA-driver 1.3.2 (from meta-intel)
>>  GStreamer 1.2.3
>>  GStreamer-VAAPI 0.5.8 (from meta-intel)
>> 
>> 
>> Dizzy 1.7.1
>>  Kernel 3.10, 3.14, 3.17 (Supportable common base)
>>  Xorg-server 1.15.1
>>  Wayland/Weston 1.5.0
>>  Xf86-video-intel 2.99.912
>>  Libdrm 2.4.54
>>  MESA 10.1.3
>>  Cairo 1.12.16
>>  libVA 1.3.1 (from meta-intel)
>>  Intel-VA-driver 1.3.2 (from meta-intel)
>>  GStreamer 1.4.1
>>  GStreamer-VAAPI 0.5.8 (from meta-intel)
>> 
>> 
>> Fido 1.8
>>  Kernel 3.14, 3.19 (supportable comon base)
>>  Xorg-server 1.16.3
>>  Wayland/weston 1.6.0
>>  Xf86-video-intel 2.99.917
>>  Libdrm 2.4.59
>>  Mesa 10.4.4
>>  Cairo 1.12.18
>>  LibVA 1.5.0 (from meta-intel)
>>  Intel-VA-driver 1.5.0 (from meta-intel)
>>  Gstreamer 1.4.5
>>  Gstreamer-vaapi 0.5.10 (from meta-intel)
>> 
>> 
>> ... Vincent
>> 
>> -Original Message-
>> From: Chang, Rebecca Swee Fun
>> Sent: Wednesday, June 10, 2015 9:08 AM
>> To: Cheah, Vincent Beng Keat
>> Cc: meta-in...@yoctoproject.org; Chris Tapp; Yocto Project; Wold, Saul; 
>> 'Paul Eggleton'
>> Subject: RE: [meta-intel] "Crazy" Xorg memory usage after upgrading from 
>> Daisy to Fido
>> 
>> Hi Vincent,
>> 
>> Can you help to comment on this issue mentioned by Chris?
>> Thanks.
>> 
>> Regards,
>> Rebecca
>> 
>>> -Original Message-
>>> From: Paul Eggleton [mailto:paul.eggle...@linux.intel.com]
>>> Sent: 09 June, 2015 12:15 AM
>>> To: Chang, Rebecca Swee Fun
>>> Cc: meta-in...@yoctoproject.org; Chris Tapp; Yocto Project; Wold, Saul
>>> Subject: Re: [meta-intel] "Crazy" Xorg memory usage after upgrading
>>> from Daisy to Fido
>>> 
>>> Rebecca, is this something you or one of your colleagues would be able
>>> to help with?
>>> 
>>> Thanks,
>>> Paul
>>> 
>>> On Friday 05 June 2015 08:29:00 Chris Tapp wrote:
 I’ve got an application that I’ve had running nicely under Daisy for
 some time. As Daisy is now a bit old, I decided to move the
 application to
>>> Fido.
 I’m using the meta-intel/isg/valleyisland BSP and also switched to
 using its Fido branch.
 
 The move only required a few minor changes and allowed me to drop a
 Daisy “updates” layer that I had been using for things like gstreamer-1.0.
 
 However, there is one behaviour which is killing me - I keep getting
 oom-killer events!
 
 The application is basically an OpenGL-ES 2.0 application that
 renders various bits of text, images and streams captured from a
 gstreamer pipeline at 60 Hz to a 1080 screen.
 
 Under Daisy this generally took just under 50% CPU and used a modest
 percentage of the 4 GB system memory - i.e. no where near running
 out and usage was just about static.
 
 Under Fido the CPU usage is about the same and the memory used by
 the application itself looks reasonable when compared to Daisy (and
 usage is static). However, the memory used by XOrg is far from
 constant or stable - it basically has a VSZ value cycling from about
 630m to 2989m with the cycle period being in the order of 5 to 10
 seconds. Peaks in XOrg memory usage coincide with stutters in 

Re: [yocto] [PSPLASH 0/2] Few fixes for psplash

[Burton, Ross]

On 24 November 2015 at 18:39, Bystricky, Juro 
wrote:

> Thanks. I’ll mark the bug 7236 as resolved/fixed.
>
>
The recipe needs updating first, just bumping the srcrev will do nicely.

Ross
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [OE-core] RFC: Reference updater filesystem

[Lopez, Mariano]

On 11/24/2015 1:32 AM, Randy Witt wrote:



On Mon, Nov 23, 2015 at 1:41 PM, Mariano Lopez 
> 
wrote:


There has been interest in an image based software updater in
Yocto Project. The proposed solution for a image based updater is
to use Stefano Babic's software updater
(http://sbabic.github.io/swupdate). This software do a binary
copy, so it is needed to have at least two partitions, these
partitions would be the rootfs and the maintenance partition. The
rootfs it's the main partition used to boot during the normal
device operation, on the other hand, the maintenance will be used
to update the main partition.

To update the system, the user has to connect to device and boot
in the maintenance partition; once in the maintenance partition
the software updater will copy the new image in the rootfs
partition. A final reboot into the rootfs it is necessary to
complete the upgrade.

As mentioned before the the software will copy an image to the
partition, so everything in that partition will be wiped out,
including custom configurations. To avoid the loss of
configuration I explore three different solutions:
1. Use a separate partition for the configuration.
  a. The pro of this method is the partition is not touched during
the update.
  b. The con of this method is the configuration is not directly
in rootfs (example: /etc).

Configuration files can be anywhere a package decides to install them. 
So having a single partition would be difficult. If you could, you 
would most likely be forced to have an initramfs to make sure /etc was 
mounted before init runs.


/etc was an example, the image should have the required files to make 
the target boot and the get the application configuration from this 
other partition. This is like openwrt does, it has a read-only rootfs 
and small read-write partition where the user can write its 
configuration and restore it at boot time.



2. Do the backup during the update.
  a. The pro is the configuration is directly in rootfs.
  b. The con is If the update fail most likely the configuration
would be lost.

Why would the configuration be lost if the update fails? Couldn't it 
just be stored on the thumbdrive?


If there is a power loss while the configuration is copied, the 
partition could go corrupt and would be difficult to recover. And as you 
mentioned  before the configuration files could be anywhere, so the 
script must be customized to get all those files and once the update is 
complete another script must restore those files, these could be 
cumbersome instead of the application have the config in another partition.



3. Have an OverlayFS for the rootfs or the partition that have the
configuration.
  a. The pro is the configuration is  "directly" in rootfs.
  b. The con is there is need to provide a custom init to
guarantee the Overlay is mounted before the boot process.

With the above information I'm proposing to use a separate
partition for the configuration; this is because is more reliable
and doesn't require big changes in the current architecture.

So, the idea is to have 4 partitions in the media:
1. boot. This is the usual boot partition
2. data. This will hold the configuration files. Not modified by
updates.
3. maintenance. This partition will be used to update rootfs.
4. rootfs. Partition used for normal operation.

Mariano
-- 
___

Openembedded-core mailing list
openembedded-c...@lists.openembedded.org

http://lists.openembedded.org/mailman/listinfo/openembedded-core




--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [PSPLASH 0/2] Few fixes for psplash

[Bystricky, Juro]

Thanks. I’ll mark the bug 7236 as resolved/fixed.
Juro


From: Burton, Ross [mailto:ross.bur...@intel.com]
Sent: Tuesday, November 24, 2015 8:58 AM
To: Bystricky, Juro
Cc: yocto@yoctoproject.org; Juro Bystricky
Subject: Re: [yocto] [PSPLASH 0/2] Few fixes for psplash


On 24 November 2015 at 16:39, Juro Bystricky 
> wrote:
These patches fix https://bugzilla.yoctoproject.org/show_bug.cgi?id=7236
Considering there was a bug in the BGR888 detection, it is probaly safe to
assume it was never used. I modified handling of 24BPP modes somewhat as well,
but did not test it. 24BPP changes only affect big endian mode, little endian
should be the same as before. (If you are aware of anybody actually using 24BPP
modes, please let me know)
Tested with jethro/core-image-sato (qemuppc, qemumips, qemuarm).

Thanks Juro.  Merged to psplash master.

Ross
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [OE-core] RFC: Reference updater filesystem

[Lopez, Mariano]

On 11/24/2015 12:06 AM, Anders Darander wrote:

* Mariano Lopez  [151123 22:41]:


There has been interest in an image based software updater in Yocto Project.

Ok. Sure, it might be nice with something that can be shared, instead
off everyone's building our own solutions.


The idea is to integrate one, not build one from the scratch.




The proposed solution for a image based updater is to use Stefano Babic's
software updater (http://sbabic.github.io/swupdate). This software do a
binary copy, so it is needed to have at least two partitions, these
partitions would be the rootfs and the maintenance partition. The rootfs
it's the main partition used to boot during the normal device operation, on
the other hand, the maintenance will be used to update the main partition.

I haven't checked the swupdate tool, though I'd suspect that it also
supports the alternating rootfs use case? (I.e. run system1 update
system2; reboot to system2. Next update is system1). This is a rather
common setup, not least when you need a remote upgrade facility.

Would your proposed inclusion to the Yocto Project support that case
too?


Yeah, it would be possible to have two "rootfs" and do the update and 
the just reboot one time.





To update the system, the user has to connect to device and boot in the
maintenance partition; once in the maintenance partition the software
updater will copy the new image in the rootfs partition. A final reboot into
the rootfs it is necessary to complete the upgrade.

Like said above, not all system can be reached manually (at least not in
cost efficient way). Sure, the mainenance partition scheme can be made
to work anyway...


I plan to release this in phases, in the first one it will be manually 
do the update. The idea is implement tools to automate the process of 
the update (where it can be automated).





As mentioned before the the software will copy an image to the partition, so
everything in that partition will be wiped out, including custom
configurations. To avoid the loss of configuration I explore three different
solutions:
1. Use a separate partition for the configuration.
   a. The pro of this method is the partition is not touched during the
update.
   b. The con of this method is the configuration is not directly in rootfs
(example: /etc).

I'd vote for that as well. Though, I only keep the re-writable
configurations here. The one that are constant between all systems are
shipped in /etc in the read-only-rootfs.


With the above information I'm proposing to use a separate partition for the
configuration; this is because is more reliable and doesn't require big
changes in the current architecture.
So, the idea is to have 4 partitions in the media:
1. boot. This is the usual boot partition
2. data. This will hold the configuration files. Not modified by updates.
3. maintenance. This partition will be used to update rootfs.
4. rootfs. Partition used for normal operation.

How flexible to you intend to make this system? Allow everything that
swupdate supports? Or a specific subset?


If you are referring to the filesystem creation I would say very 
flexible. It will be implemented using wic instead of a class, so just 
needs to change a file to suit your needs. If you refer to the swupdate 
features, I plan to have a generic use case; as an example I won't use 
the MTD capabilities of the software.




Cheers,
Anders



--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [OE-core] [oe] RFC: Reference updater filesystem

[Mark Hatle]

On 11/24/15 11:02 AM, Lopez, Mariano wrote:
> 
> 
> On 11/24/2015 7:47 AM, Mark Hatle wrote:
>> On 11/24/15 4:39 AM, Roman Khimov wrote:
>>> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал:
 1. Use a separate partition for the configuration.
 a. The pro of this method is the partition is not touched during the
 update.
 b. The con of this method is the configuration is not directly in
 rootfs (example: /etc).
>>> That's the right solution, although to do it really right (at least IMO) you
>>> need to implement the /usr merge [1] (and that's orthogonal to using or not
>>> using systemd), which can also help you make your /usr read-only (because
>>> that's just code and static data) with read-write / for user data of various
>>> sorts.
>> Why does merging /usr have anything to do with this?  I've read the case for
>> merging /usr and / and still don't understand why it "helps".  The key is 
>> that
>> if you have separate partitions for /usr and /, then you need to update both 
>> of
>> them in sequence.  Merging these two just seems like a lazy solution to 
>> people
>> not wanting to deal with early boot being self-contained.
>>
>> Also having a separate / from /usr can help with '/' be your maintenance
>> partition in some cases.
>>
 3. Have an OverlayFS for the rootfs or the partition that have the
 configuration.
 a. The pro is the configuration is  "directly" in rootfs.
 b. The con is there is need to provide a custom init to guarantee the
 Overlay is mounted before the boot process.
>>> And this is the approach I would recommend not doing. I've used UnionFS for
>>> thing like that (overlaying whole root file system) some 6 years ago, it
>>> sounded nice and it kinda worked, but it wasn't difficult to make it fail
>>> (just a little playing with power), we've even seen failures on production
>>> devices, like when you have whiteout file for directory already written, but
>>> don't have new files in it yet and that can completely ruin the system.
>>>
>>> Also, it usually works better when you don't have any changes in the lower
>>> layer, but we're talking about updating it here, you can easily end up in a
>>> situation where you have updated something in the rootfs but that was
>>> overriden by upper layer and thus your user doesn't see any change.
>> When using overlayfs, I'd strongly recommend not doing it over the entire
>> rootfs.  This is generally a bad idea for the reasons stated above.
>>
>> However, overlaying a part of the rootfs often makes sense.  /etc is a good
>> example.  This way applications that want their configurations in /etc can 
>> still
>> have it that way -- and there is always a (hopefully) reasonable default
>> configuration, should the configuration 'partition' get corrupted.  So worst
>> case the user can start over on configurations only.
> 
> Do you know a way to mount the overlay before all the services start? I 
> tried to do this, but the only reliable way to do it was using a custom 
> init, I couldn't accomplish this using systemd or sysvnit.

In the past I've done this with an initrd, with a custom /sbin/init that mounted
and then did a reexec for the real init system or ordered things in such a way
that the overlay happened -very- early.

>>
>> For applications and user data, these can and should be stored outside of the
>> main rootfs.  The FHS/LSB recomment '/opt', but while it doesn't matter if 
>> it's
>> -actually- /opt, the concept itself is good.
>>
>>
>> So going back to image upgrade.  The key here is that you need a way to 
>> update
>> arbitrary images with arbitrary contents and a mechanisms that is smart 
>> enough
>> to generate the update (vs a full image flash) to conserve bandwidth.
> 
> I was concerned about this too, not just bandwidth but resources in the 
> target. Unfortunately I couldn't find an option that is generic enough 
> to just provide the update. The idea is to integrate the tool into YP, 
> not to develop a new one. Some of the tools that I checked needed to use 
> btrfs partitions, need python in the target, or other constrains that 
> make the update system impossible for a lot of targets.

Yup.  I just want to make sure people know one tool isn't going to do
everything..  and the integration of a single tool shouldn't restrict people for
doing other things with custom tooling.

--Mark

>>
>> I still contend it's up to the device to be able to configure the system on 
>> how
>> to get the update and where to apply the update.  The tooling (host and 
>> target)
>> should simply assist with this.
>>
>> Delta updates need version information in order to know they're doing the 
>> right
>> sequence of updating.
>>
>> Full updates don't, but should be sent in a format that limits "empty space",
>> effectively send them as sparse files.
>>
>> On many devices you will need to flash as part of the download due to space
>> limitations.
> 
> The tool mentioned 

Re: [yocto] [OE-core] [oe] RFC: Reference updater filesystem

[Lopez, Mariano]

On 11/24/2015 7:47 AM, Mark Hatle wrote:

On 11/24/15 4:39 AM, Roman Khimov wrote:

В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал:

1. Use a separate partition for the configuration.
a. The pro of this method is the partition is not touched during the
update.
b. The con of this method is the configuration is not directly in
rootfs (example: /etc).

That's the right solution, although to do it really right (at least IMO) you
need to implement the /usr merge [1] (and that's orthogonal to using or not
using systemd), which can also help you make your /usr read-only (because
that's just code and static data) with read-write / for user data of various
sorts.

Why does merging /usr have anything to do with this?  I've read the case for
merging /usr and / and still don't understand why it "helps".  The key is that
if you have separate partitions for /usr and /, then you need to update both of
them in sequence.  Merging these two just seems like a lazy solution to people
not wanting to deal with early boot being self-contained.

Also having a separate / from /usr can help with '/' be your maintenance
partition in some cases.


3. Have an OverlayFS for the rootfs or the partition that have the
configuration.
a. The pro is the configuration is  "directly" in rootfs.
b. The con is there is need to provide a custom init to guarantee the
Overlay is mounted before the boot process.

And this is the approach I would recommend not doing. I've used UnionFS for
thing like that (overlaying whole root file system) some 6 years ago, it
sounded nice and it kinda worked, but it wasn't difficult to make it fail
(just a little playing with power), we've even seen failures on production
devices, like when you have whiteout file for directory already written, but
don't have new files in it yet and that can completely ruin the system.

Also, it usually works better when you don't have any changes in the lower
layer, but we're talking about updating it here, you can easily end up in a
situation where you have updated something in the rootfs but that was
overriden by upper layer and thus your user doesn't see any change.

When using overlayfs, I'd strongly recommend not doing it over the entire
rootfs.  This is generally a bad idea for the reasons stated above.

However, overlaying a part of the rootfs often makes sense.  /etc is a good
example.  This way applications that want their configurations in /etc can still
have it that way -- and there is always a (hopefully) reasonable default
configuration, should the configuration 'partition' get corrupted.  So worst
case the user can start over on configurations only.


Do you know a way to mount the overlay before all the services start? I 
tried to do this, but the only reliable way to do it was using a custom 
init, I couldn't accomplish this using systemd or sysvnit.




For applications and user data, these can and should be stored outside of the
main rootfs.  The FHS/LSB recomment '/opt', but while it doesn't matter if it's
-actually- /opt, the concept itself is good.


So going back to image upgrade.  The key here is that you need a way to update
arbitrary images with arbitrary contents and a mechanisms that is smart enough
to generate the update (vs a full image flash) to conserve bandwidth.


I was concerned about this too, not just bandwidth but resources in the 
target. Unfortunately I couldn't find an option that is generic enough 
to just provide the update. The idea is to integrate the tool into YP, 
not to develop a new one. Some of the tools that I checked needed to use 
btrfs partitions, need python in the target, or other constrains that 
make the update system impossible for a lot of targets.




I still contend it's up to the device to be able to configure the system on how
to get the update and where to apply the update.  The tooling (host and target)
should simply assist with this.

Delta updates need version information in order to know they're doing the right
sequence of updating.

Full updates don't, but should be sent in a format that limits "empty space",
effectively send them as sparse files.

On many devices you will need to flash as part of the download due to space
limitations.


The tool mentioned has this capability.



And you need the ability to flash multiple partitions.

maintenance
/
/usr
data

etc..  whatever it takes to either upgrade or restore the device.


Yes, that would be possible, the only limitation is that is not possible 
to flash the partition that is being used.




--Mark


With the above information I'm proposing to use a separate partition for
the configuration; this is because is more reliable and doesn't require
big changes in the current architecture.

So, the idea is to have 4 partitions in the media:
1. boot. This is the usual boot partition
2. data. This will hold the configuration files. Not modified by updates.
3. maintenance. This partition will be used to update rootfs.
4. rootfs. 

Re: [linux-yocto] [PULL REQUEST] Intel Axxia updates to linux-yocto-4.1

[Bruce Ashfield]

On 15-11-19 11:55 AM, Bruce Ashfield wrote:

On 15-11-19 11:31 AM, Cristian Bercaru wrote:

Hello!

This series of patches brings various improvements to the Intel Axxia
drivers from linux-yocto-4.1, including NCR, PCIe, Dickens, amba-pl011
and XLF device trees.

Please pull the patches from
https://github.com/lsigithub/axxia_yocto_linux_4.1_pull.git
into git://git.yoctoproject.org/linux-yocto-4.1
Pull
'standard/axxia/base-1.18' -> 'standard/axxia/base'
'standard/preempt-rt/axxia/base-1.18' -> 'standard/preempt-rt/axxia/base'

If you add more patches to standard/(preempt-rt)/axxia/base beforehand
please notify me. I will rebase our changes so you can do a clean,
fast-forward pull.


I can notify you in advance! I'm right now updating linux-yocto 4.1
to the latest -stable, and refreshing -rt.

I can send another email when that is done, so you'll know to rebase.


Sorry for the slow reply, the -rt updated caused some issues.

I don't have my SRCREV updates out yet, but if you update your 
linux-yocto-4.1

tree, you'll see the new 4.1 and -rt updates.

So rebase and resend and we'll be good to merge.

Bruce



Bruce



Thank you,
Cristian Bercaru

Dave Martin (1):
   serial/amba-pl011: Refactor and simplify TX FIFO handling

Gary McGee (1):
   arch/arm/mach-axxia : Make AXXIA_NCR_RESET_CHECK a Kconfig Option

John Jacques (6):
   arch/arm64: Axxia Device Tree Update for XLF
   drivers/misc: Clean Up and Improve the Axxia NCR Driver
   arch/arm/kvm: KVM_VTTBR Is Too Large for 'add'
   drivers/misc: Clean Up the NCR Driver
   drivers/misc/lsi-ncr.c: Move the Header to the Default Path
   arch/arm/mach-axxia: Clean Up the Mapping of Dickens

SangeethaRao (2):
   drivers/pci/host: Updated X7 PCIe Driver
   drivers/pci/host: Updated Kconfig

Sreedevi Joshi (2):
   char: hwrng: Enabled trng driver for XLF
   misc: mtc: Enable XLF MTC driver

  arch/arm/boot/dts/axm55xx.dtsi|   4 +-
  arch/arm/include/asm/lsi/acp_ncr.h|  45 
  arch/arm/kvm/interrupts.S |   6 +-
  arch/arm/mach-axxia/Kconfig   |   7 +
  arch/arm/mach-axxia/axxia.c   |  53 ++--
  arch/arm/mach-axxia/axxia.h   |   2 +
  arch/arm/mach-axxia/ddr_retention.c   | 186 ++---
  arch/arm/mach-axxia/include/mach/ncr.h|  44 ---
  arch/arm/mach-axxia/perf_event_platform.c |   2 +-
  arch/arm/mach-axxia/platsmp.c |  29 +-
  arch/arm/mach-axxia/rapidio.c |   2 +-
  arch/arm/mach-axxia/smon.c|   3 +-
  arch/arm64/boot/dts/intel/axc6704-emu.dts |   8 +
  arch/arm64/boot/dts/intel/axc6712-emu.dts |  10 +
  arch/arm64/boot/dts/intel/axc67xx.dtsi|   2 +-
  drivers/edac/axxia_edac-l2_cpu.c  |   2 +-
  drivers/edac/axxia_edac-l3.c  |   2 +-
  drivers/edac/axxia_edac-mc.c  |   2 +-
  drivers/misc/lsi-mtc.c|  48 +++-
  drivers/misc/lsi-ncr.c| 435
--
  drivers/misc/lsi-ncr.h|  42 ---
  drivers/net/ethernet/lsi/lsi_acp_net.c|   3 +-
  drivers/pci/host/Kconfig  |   2 +-
  drivers/pci/host/axxia_pci.c  |  24 +-
  drivers/tty/serial/amba-pl011.c   | 113 ++--
  include/linux/lsi-ncr.h   |  55 
  include/linux/lsi_mtc_ioctl.h |  87 +-
  27 files changed, 665 insertions(+), 553 deletions(-)
  delete mode 100644 arch/arm/include/asm/lsi/acp_ncr.h
  delete mode 100644 arch/arm/mach-axxia/include/mach/ncr.h
  delete mode 100644 drivers/misc/lsi-ncr.h
  create mode 100644 include/linux/lsi-ncr.h





--
___
linux-yocto mailing list
linux-yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/linux-yocto

[yocto] Weston Compositor on Yocto core-image-weston-beaglebone

[Gerard Mc Cabe]

Hi,

Im trying to run the weston compositor on a beaglebone black with yocto
core-image-weston. Its failing when trying to load the gl-renderer.so
module. Please see below the terminal output.

I've searched online but there’s not much available on this subject. Any
advice would be appreciated.

Thanks,

Gerard.



Poky (Yocto Project Reference Distro) 1.6.3 beaglebone /dev/ttyO0

beaglebone login: root
root@beaglebone:~# mkdir -p /tmp/$USER-weston
root@beaglebone:~# chmod 0700 /tmp/$USER-weston
root@beaglebone:~# export XDG_RUNTIME_DIR=/tmp/$USER-weston
root@beaglebone:~# weston --tty=1
Date: 2015-11-24 UTC
[15:37:47.672] weston 1.4.0
   http://wayland.freedesktop.org/
   Bug reports to:
https://bugs.freedesktop.org/enter_bug.cgi?product=Wayland=weston=1.4.0
   Build: 1.4.0-dirty configure.ac: Bump version to 1.4.0
(2014-01-23 20:51:40 -0800)
[15:37:47.673] OS: Linux, 3.14.19-yocto-standard, #1 PREEMPT Thu Nov 19
13:48:10 GMT 2015, armv7l
[15:37:47.674] Starting with no config file.
[15:37:47.674] Loading module '/usr/lib/weston/drm-backend.so'
[15:37:47.683] initializing drm backend
[15:37:47.687] using /dev/dri/card0
[15:37:47.687] Loading module '/usr/lib/weston/gl-renderer.so'
gbm: malformed or no PCI IDfailed to load module:
/usr/lib/gbm/gbm_gallium_drm.so: cannot open shared object file: No such
file or directory
[15:37:47.690] failed to initialize egl
[15:37:47.696] fatal: failed to create compositor
root@beaglebone:~#
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Header files in /usr/include of rootfs

[洪翊桓_Pegatron]

Hi there,

I was trying to add some pre-built binaries and files by using a recipe.
The recipe has something like this:

SRC_URI = file://files
…
do_install() {
install –d ${D}
}

FILES_${PN} = “/*”
ALLOW_EMPTY_${PN} = “1”
INSANE_SKIP_${PN} = “debug-files dev-so”

The result was all the binaries and files installed and packages are created,
But I also found kernel header files are also built in under /usr/include in 
rootfs image.
For my understanding those header files should be used if I want to 
develop/debug applications on the target device but it is not the case.

Also I noticed the use of INSANE_SKIP_${PN} so I check the usage of debug-files 
and dev-so,
And it turned out there are .debug folders in the WORKDIR/package folder so I 
thought that is why debug-files was used.

The questions are,
Why are those kernel header files installed in rootfs? How do I prevent this?
Why are there .debug in the package folder, it should only be in package –dbg 
folder, right?

I searched and figured maybe it has something to do with the do_install task 
but I can’t find a solution.
Any suggestions?

Thanks,
Andy
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Read-only rootfs update

[Ruben Schwarz]

Hi,

We are using yocto to build a custom linux image for our BeagleBoneBlack
based hardware. We are using a read-only root file system to prevent data
lost if power supply is lost.
Now we wan't to be able to update our device. If it is possiple we wan't to
use apt/dpkg to do this update. We set up an update server and everything
works so far.

Following steps are done to do an update:

mount -o remount,rw /
apt-get update
apt-get upgrade

Everything works fine, new package versions are installed. After rebooting
the device and executing "dpkg -l" the old package version is listed.
Output of mount shows that there is tmpfs on /var/lib which hides the
/var/lib/dpkg folder.


root@device:~# mount
rootfs on / type rootfs (rw)
/dev/root on / type ext4 (ro,relatime,data=ordered)
devtmpfs on /dev type devtmpfs
(rw,relatime,size=243140k,nr_inodes=60785,mode=755)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/volatile type tmpfs (rw,relatime)
tmpfs on /var/lib type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)




Is there an standard way how to update devices with read-only root file
systems? Anybody here who did something similar?

Best regards
Ruben

-- 
Ruben Schwarz | SOTEC | r.schw...@sotec.eu | T +49 7033 5458 34 | F +49
7033 5458 95

-- 


Besuchen Sie uns auf der SPS IPC Drives vom 24. - 26.11. in Nürnberg 

Ihr kostenloser Gutscheincode: 1551219857

Sichern Sie sich heute noch einen Platz bei Google's Online Event über den 
Arbeitsplatz der Zukunft 

www.sotec.eu | www.cloudwuerdig.com 
-- 
SOTEC Software Entwicklungs GmbH + Co Mikrocomputertechnik KG 
Calwer Straße 11, D-75395 Ostelsheim 
Sitz Ostelsheim, Amtsgericht Stuttgart HRA 330821/HRB 330664, 
Geschäftsführer: Florian Holz 
Cloudwürdig® ist eine eingetragene Marke der SOTEC Software Entwicklungs 
GmbH + Co Mikrocomputertechnik KG

Der Inhalt dieser e-Mail ist ausschließlich für den bezeichneten Adressaten 
bestimmt. Wenn Sie nicht der vorgesehene Adressat dieser e-Mail oder dessen 
Vertreter sein sollten, so beachten Sie bitte, dass jede Form der 
Kenntnisnahme, Veröffentlichung, Vervielfältigung oder Weitergabe des 
Inhalts dieser e-Mail unzulässig ist. Wir bitten Sie, sich in diesem Fall 
mit dem Absender der e-Mail in Verbindung zu setzen.

The content of this e-mail is meant exclusively for the person to whom it 
is addressed. If you are not the person to whom this e-mail is addressed or 
his/her representative, please be informed, that any form of knowledge, 
publication, duplication or distribution of the content of this e-mail is 
inadmissible. We ask you, therefore, in such a case to please contact the 
sender of this e-mail.

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [OE-core] [oe] RFC: Reference updater filesystem

[Mark Hatle]

On 11/24/15 4:39 AM, Roman Khimov wrote:
> В письме от 23 ноября 2015 15:41:28 пользователь Mariano Lopez написал:
>> 1. Use a separate partition for the configuration.
>>a. The pro of this method is the partition is not touched during the
>> update.
>>b. The con of this method is the configuration is not directly in
>> rootfs (example: /etc).
> 
> That's the right solution, although to do it really right (at least IMO) you 
> need to implement the /usr merge [1] (and that's orthogonal to using or not 
> using systemd), which can also help you make your /usr read-only (because 
> that's just code and static data) with read-write / for user data of various 
> sorts.

Why does merging /usr have anything to do with this?  I've read the case for
merging /usr and / and still don't understand why it "helps".  The key is that
if you have separate partitions for /usr and /, then you need to update both of
them in sequence.  Merging these two just seems like a lazy solution to people
not wanting to deal with early boot being self-contained.

Also having a separate / from /usr can help with '/' be your maintenance
partition in some cases.

>> 3. Have an OverlayFS for the rootfs or the partition that have the
>> configuration.
>>a. The pro is the configuration is  "directly" in rootfs.
>>b. The con is there is need to provide a custom init to guarantee the
>> Overlay is mounted before the boot process.
> 
> And this is the approach I would recommend not doing. I've used UnionFS for 
> thing like that (overlaying whole root file system) some 6 years ago, it 
> sounded nice and it kinda worked, but it wasn't difficult to make it fail 
> (just a little playing with power), we've even seen failures on production 
> devices, like when you have whiteout file for directory already written, but 
> don't have new files in it yet and that can completely ruin the system.
> 
> Also, it usually works better when you don't have any changes in the lower 
> layer, but we're talking about updating it here, you can easily end up in a 
> situation where you have updated something in the rootfs but that was 
> overriden by upper layer and thus your user doesn't see any change.

When using overlayfs, I'd strongly recommend not doing it over the entire
rootfs.  This is generally a bad idea for the reasons stated above.

However, overlaying a part of the rootfs often makes sense.  /etc is a good
example.  This way applications that want their configurations in /etc can still
have it that way -- and there is always a (hopefully) reasonable default
configuration, should the configuration 'partition' get corrupted.  So worst
case the user can start over on configurations only.

For applications and user data, these can and should be stored outside of the
main rootfs.  The FHS/LSB recomment '/opt', but while it doesn't matter if it's
-actually- /opt, the concept itself is good.


So going back to image upgrade.  The key here is that you need a way to update
arbitrary images with arbitrary contents and a mechanisms that is smart enough
to generate the update (vs a full image flash) to conserve bandwidth.

I still contend it's up to the device to be able to configure the system on how
to get the update and where to apply the update.  The tooling (host and target)
should simply assist with this.

Delta updates need version information in order to know they're doing the right
sequence of updating.

Full updates don't, but should be sent in a format that limits "empty space",
effectively send them as sparse files.

On many devices you will need to flash as part of the download due to space
limitations.

And you need the ability to flash multiple partitions.

maintenance
/
/usr
data

etc..  whatever it takes to either upgrade or restore the device.

--Mark

>> With the above information I'm proposing to use a separate partition for
>> the configuration; this is because is more reliable and doesn't require
>> big changes in the current architecture.
>>
>> So, the idea is to have 4 partitions in the media:
>> 1. boot. This is the usual boot partition
>> 2. data. This will hold the configuration files. Not modified by updates.
>> 3. maintenance. This partition will be used to update rootfs.
>> 4. rootfs. Partition used for normal operation.
> 
> You probably don't need to separate 1 and 3, all the code for system update 
> should easily fit into initramfs and just making /boot a bit larger would 
> allow you to store some backup rootfs.
> 
> Also, you can swap 4 and 2 which will be useful if you're installing on 
> different sized storage devices, usually you know good enough the size of 
> your 
> rootfs, but you probably want to leave more space for user data if there is 
> an 
> opportunity to do so, that's just easier to do with data partition at the end.
> 
> [1] http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
> 

-- 
___
yocto mailing list
yocto@yoctoproject.org