Re: [OE-core] [PATCH] externalsrc: Detect code changes in submodules

2021-02-08 Thread Douglas via lists.openembedded.org 

On 6/02/21 1:01 pm, Jon Mason wrote:

On Thu, Feb 4, 2021 at 10:43 PM Douglas via lists.openembedded.org
 wrote:

Further to 50ff9afb39, only detect code changes in submodules that are
subdirectories of the EXTERNALSRC directory.

Signed-off-by: Douglas Royds 
---

I believe this patch is causing errors in the autobuilder.  See
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1801/steps/14/logs/stdio



Good find by the autobuilder. Earning its money.

This is a git weakness: In this case of a local-files-only recipe, 
devtool creates an empty git repository and `git submodule--helper list 
path/` fails. Add a file (not even committed), and it works just fine.


Updated patch on its way.





  meta/classes/externalsrc.bbclass | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 64e94e3301..27496bb2db 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -216,7 +216,7 @@ def srctree_hash_files(d, srcdir=None):
  env['GIT_INDEX_FILE'] = tmp_index.name
  subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, 
env=env)
  git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, 
env=env).decode("utf-8")
-submodule_helper = subprocess.check_output(['git', 'submodule--helper', 
'list'], cwd=s_dir, env=env).decode("utf-8")
+submodule_helper = subprocess.check_output(['git', 'submodule--helper', 
'list', s_dir], cwd=s_dir, env=env).decode("utf-8")
  for line in submodule_helper.splitlines():
  module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
  proc = subprocess.Popen(['git', 'add', '-A', '.'], 
cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
--
2.25.1






--
This Communication is Confidential. We only send and receive email on the
basis of the terms set out at www.taitradio.com/email_disclaimer 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#147849): 
https://lists.openembedded.org/g/openembedded-core/message/147849
Mute This Topic: https://lists.openembedded.org/mt/80398669/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] externalsrc: Detect code changes in submodules

2021-02-05 Thread Scott Branden via lists.openembedded.org 
This patch, now integrated in master branch of poky, appears to also fix 
this yocto bug:


"Bug 13748 - bitbake doesn't detect changes in code to run do_compile 
when using devtool modify on recipe with destsuffix"

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13748

On 2021-01-27 12:33 a.m., Tomasz Dziendzielski wrote:

The srctree_hash was calculated only from main source directory ignoring
changes in submodules.

[YOCTO #13748]

Use submodule--helper to determine all submodules, and calculate hash
from all git tree objects names combined.

Signed-off-by: Tomasz Dziendzielski 
---
  meta/classes/externalsrc.bbclass | 12 +++-
  1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 7a7d31e311..64e94e3301 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -190,6 +190,7 @@ def srctree_hash_files(d, srcdir=None):
  import shutil
  import subprocess
  import tempfile
+import hashlib
  
  s_dir = srcdir or d.getVar('EXTERNALSRC')

  git_dir = None
@@ -214,7 +215,16 @@ def srctree_hash_files(d, srcdir=None):
  env = os.environ.copy()
  env['GIT_INDEX_FILE'] = tmp_index.name
  subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, 
env=env)
-sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, 
env=env).decode("utf-8")
+git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, 
env=env).decode("utf-8")
+submodule_helper = subprocess.check_output(['git', 'submodule--helper', 
'list'], cwd=s_dir, env=env).decode("utf-8")
+for line in submodule_helper.splitlines():
+module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
+proc = subprocess.Popen(['git', 'add', '-A', '.'], 
cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+proc.communicate()
+proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, 
env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+stdout, _ = proc.communicate()
+git_sha1 += stdout.decode("utf-8")
+sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
  with open(oe_hash_file, 'w') as fobj:
  fobj.write(sha1)
  ret = oe_hash_file + ':True'









smime.p7s
Description: S/MIME Cryptographic Signature

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#147719): 
https://lists.openembedded.org/g/openembedded-core/message/147719
Mute This Topic: https://lists.openembedded.org/mt/80152944/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] externalsrc: Detect code changes in submodules

2021-02-05 Thread Jon Mason 
On Thu, Feb 4, 2021 at 10:43 PM Douglas via lists.openembedded.org
 wrote:
>
> Further to 50ff9afb39, only detect code changes in submodules that are
> subdirectories of the EXTERNALSRC directory.
>
> Signed-off-by: Douglas Royds 
> ---

I believe this patch is causing errors in the autobuilder.  See
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1801/steps/14/logs/stdio

>  meta/classes/externalsrc.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/externalsrc.bbclass 
> b/meta/classes/externalsrc.bbclass
> index 64e94e3301..27496bb2db 100644
> --- a/meta/classes/externalsrc.bbclass
> +++ b/meta/classes/externalsrc.bbclass
> @@ -216,7 +216,7 @@ def srctree_hash_files(d, srcdir=None):
>  env['GIT_INDEX_FILE'] = tmp_index.name
>  subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, 
> env=env)
>  git_sha1 = subprocess.check_output(['git', 'write-tree'], 
> cwd=s_dir, env=env).decode("utf-8")
> -submodule_helper = subprocess.check_output(['git', 
> 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
> +submodule_helper = subprocess.check_output(['git', 
> 'submodule--helper', 'list', s_dir], cwd=s_dir, env=env).decode("utf-8")
>  for line in submodule_helper.splitlines():
>  module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
>  proc = subprocess.Popen(['git', 'add', '-A', '.'], 
> cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
> --
> 2.25.1
>
>
> 
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#147713): 
https://lists.openembedded.org/g/openembedded-core/message/147713
Mute This Topic: https://lists.openembedded.org/mt/80398669/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] externalsrc: Detect code changes in submodules

2021-02-04 Thread Douglas via lists.openembedded.org 
Further to 50ff9afb39, only detect code changes in submodules that are
subdirectories of the EXTERNALSRC directory.

Signed-off-by: Douglas Royds 
---
 meta/classes/externalsrc.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 64e94e3301..27496bb2db 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -216,7 +216,7 @@ def srctree_hash_files(d, srcdir=None):
 env['GIT_INDEX_FILE'] = tmp_index.name
 subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, 
env=env)
 git_sha1 = subprocess.check_output(['git', 'write-tree'], 
cwd=s_dir, env=env).decode("utf-8")
-submodule_helper = subprocess.check_output(['git', 
'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
+submodule_helper = subprocess.check_output(['git', 
'submodule--helper', 'list', s_dir], cwd=s_dir, env=env).decode("utf-8")
 for line in submodule_helper.splitlines():
 module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
 proc = subprocess.Popen(['git', 'add', '-A', '.'], 
cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-- 
2.25.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#147679): 
https://lists.openembedded.org/g/openembedded-core/message/147679
Mute This Topic: https://lists.openembedded.org/mt/80398669/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] externalsrc: Detect code changes in submodules

2021-01-27 Thread Tomasz Dziendzielski 
The srctree_hash was calculated only from main source directory ignoring
changes in submodules.

[YOCTO #13748]

Use submodule--helper to determine all submodules, and calculate hash
from all git tree objects names combined.

Signed-off-by: Tomasz Dziendzielski 
---
 meta/classes/externalsrc.bbclass | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 7a7d31e311..64e94e3301 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -190,6 +190,7 @@ def srctree_hash_files(d, srcdir=None):
 import shutil
 import subprocess
 import tempfile
+import hashlib
 
 s_dir = srcdir or d.getVar('EXTERNALSRC')
 git_dir = None
@@ -214,7 +215,16 @@ def srctree_hash_files(d, srcdir=None):
 env = os.environ.copy()
 env['GIT_INDEX_FILE'] = tmp_index.name
 subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, 
env=env)
-sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, 
env=env).decode("utf-8")
+git_sha1 = subprocess.check_output(['git', 'write-tree'], 
cwd=s_dir, env=env).decode("utf-8")
+submodule_helper = subprocess.check_output(['git', 
'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
+for line in submodule_helper.splitlines():
+module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
+proc = subprocess.Popen(['git', 'add', '-A', '.'], 
cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+proc.communicate()
+proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, 
env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+stdout, _ = proc.communicate()
+git_sha1 += stdout.decode("utf-8")
+sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
 with open(oe_hash_file, 'w') as fobj:
 fobj.write(sha1)
 ret = oe_hash_file + ':True'
-- 
2.30.0


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#147337): 
https://lists.openembedded.org/g/openembedded-core/message/147337
Mute This Topic: https://lists.openembedded.org/mt/80152944/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-