Re: Yara not using global rules

[Víctor Manuel Álvarez García]

If what you want is preventing large files from being scanned at all, yes,
that's something you won't achieve with a global rule with a "filesize < X"
condition. People tend to think that a condition like "false and $a" means
that string $a is not searched at all, but that's not the case because as
Wesley said all strings are scanned first and the conditions are evaluated
after that.

On Thu, Aug 17, 2017 at 1:10 PM, necrophcodr  wrote:

> Ah, so the only way to avoid scanning large files, is to create a filelist
> beforehand, then remove the entires that are too large? And the same goes
> for mime types too, I guess?
>
> Den torsdag den 17. august 2017 kl. 12.55.04 UTC+2 skrev Wesley Shields:
>>
>> Ok, so it isn’t a question of incorrect matching like you originally
>> said. It is really a question of searching a file even if it doesn’t pass
>> the global rules. This is a misunderstanding I sometimes see. All strings
>> are collected and searched for in a single pass, then global rules are
>> evaluated.
>>
>> — WXS
>>
>> On Thu, Aug 17, 2017 at 6:49 AM necrophcodr  wrote:
>>
>>> Alright, I've solved the issue:
>>>
>>> Albeit this is synthetic, running
>>>
>>> ```
>>> for f in $(seq 0 100); do printf "\n\n\n\n\n\n\n\n\n\n" >> text.txt;
>>> done
>>> ```
>>>
>>> And then
>>>
>>> ```
>>> yara inc.yar .
>>> ```
>>>
>>> In the directory with the yara files, yields, on my test system:
>>>
>>> ./misc.yar(9): warning: $newline is slowing down scanning (critical!)
>>> fsL ./inc.yar
>>> fsL ./global.yar
>>> fsL ./misc.yar
>>> error scanning ./text.txt: internal error: 30
>>>
>>>
>>> And while using the newline scan is not a great idea, clearly the file
>>> is still being scanned in one way or another, in spite of the global rule.
>>>
>>>
>>> Den torsdag den 17. august 2017 kl. 12.24.11 UTC+2 skrev necrophcodr:

 I'm afraid I cannot post the exact files. I'll create a working
 environment that replicates all the variables required, and I'll post it
 here when I've gotten this done.

 Den onsdag den 16. august 2017 kl. 16.31.35 UTC+2 skrev Wesley Shields:
>
> I still can not replicate your problem.
>
> A couple of things to note however:
>
> "internal error: 30" is because there are too many matches, which
> happens when a single string matches too many times. It has nothing to do
> with file size like you guessed.
>
> Your "newline_one" rule is marked as private so it should never be
> reported.
>
> At this point I can not replicate your problem so I'm curious if you
> could zip up the files you're using to do this and post them somewhere for
> me to see them exactly? I would need your exact YARA rules and the file 
> you
> are scanning.
>
> -- WXS
>
> > On Aug 16, 2017, at 5:51 AM, necrophcodr 
> wrote:
> >
> > Alright, so I've returned with a result:
> >
> > If I have `~/inc.yar` with the following content:
> >
> > ```
> > include "./global.yar"
> > include "./misc.yar"
> > ```
> >
> > And the content of these files respectively:
> >
> > ```
> > global rule fsL { condition: filesize < 8MB }
> > ```
> >
> > And
> >
> > ```
> > private rule newline_one {
> > meta:
> > description = "Files that contain one newline"
> > author = "Steffen Rytter Postas"
> >
> > strings:
> > $newline = "\n"
> >
> > condition:
> > ( #newline == 1 )
> > }
> > ```
> >
> > Then the issue prevails.
> >
> > Note that this requires an actually large file that contains
> newlines. Doing `dd if=/dev/zero bs=4M count=250 of=file.bin` and scanning
> that won't yield usable results.
> >
> > Den onsdag den 16. august 2017 kl. 11.43.17 UTC+2 skrev necrophcodr:
> > Hi Wesley,
> >
> > Sorry for the late reply, vacations and all.
> >
> > So first and foremost:
> >
> > `yara -v`
> > yara 3.5.0
> >
> > The files getting scanned are reporting ` internal error: 30` which
> I'm reading to be due to files being too large. These files are often
> larger than 500MB too, well above the 8MB margin.
> >
> > I've attempted to replicate it using my own instructions, coupled
> with your misc.yar, and the result is that it works just fine.
> >
> > So I'm guessing the issue is with my own setup, and I'll continue
> evaluating the specifics and return with a response when I've found the
> culprit.
> >
> > Den mandag den 7. august 2017 kl. 16.06.59 UTC+2 skrev Wesley
> Shields:
> > I can't replicate this behavior using 3.5.0 or latest master.
> >
> > wxs@wxs-mbp yara % cat foo
> > include "./global.yar"
> > include "./misc.yar"
> > wxs@wxs-mbp 

Re: Yara not using global rules

[Wesley Shields]

Ok, so it isn’t a question of incorrect matching like you originally said.
It is really a question of searching a file even if it doesn’t pass the
global rules. This is a misunderstanding I sometimes see. All strings are
collected and searched for in a single pass, then global rules are
evaluated.

— WXS

On Thu, Aug 17, 2017 at 6:49 AM necrophcodr  wrote:

> Alright, I've solved the issue:
>
> Albeit this is synthetic, running
>
> ```
> for f in $(seq 0 100); do printf "\n\n\n\n\n\n\n\n\n\n" >> text.txt;
> done
> ```
>
> And then
>
> ```
> yara inc.yar .
> ```
>
> In the directory with the yara files, yields, on my test system:
>
> ./misc.yar(9): warning: $newline is slowing down scanning (critical!)
> fsL ./inc.yar
> fsL ./global.yar
> fsL ./misc.yar
> error scanning ./text.txt: internal error: 30
>
>
> And while using the newline scan is not a great idea, clearly the file is
> still being scanned in one way or another, in spite of the global rule.
>
>
> Den torsdag den 17. august 2017 kl. 12.24.11 UTC+2 skrev necrophcodr:
>>
>> I'm afraid I cannot post the exact files. I'll create a working
>> environment that replicates all the variables required, and I'll post it
>> here when I've gotten this done.
>>
>> Den onsdag den 16. august 2017 kl. 16.31.35 UTC+2 skrev Wesley Shields:
>>>
>>> I still can not replicate your problem.
>>>
>>> A couple of things to note however:
>>>
>>> "internal error: 30" is because there are too many matches, which
>>> happens when a single string matches too many times. It has nothing to do
>>> with file size like you guessed.
>>>
>>> Your "newline_one" rule is marked as private so it should never be
>>> reported.
>>>
>>> At this point I can not replicate your problem so I'm curious if you
>>> could zip up the files you're using to do this and post them somewhere for
>>> me to see them exactly? I would need your exact YARA rules and the file you
>>> are scanning.
>>>
>>> -- WXS
>>>
>>> > On Aug 16, 2017, at 5:51 AM, necrophcodr  wrote:
>>> >
>>> > Alright, so I've returned with a result:
>>> >
>>> > If I have `~/inc.yar` with the following content:
>>> >
>>> > ```
>>> > include "./global.yar"
>>> > include "./misc.yar"
>>> > ```
>>> >
>>> > And the content of these files respectively:
>>> >
>>> > ```
>>> > global rule fsL { condition: filesize < 8MB }
>>> > ```
>>> >
>>> > And
>>> >
>>> > ```
>>> > private rule newline_one {
>>> > meta:
>>> > description = "Files that contain one newline"
>>> > author = "Steffen Rytter Postas"
>>> >
>>> > strings:
>>> > $newline = "\n"
>>> >
>>> > condition:
>>> > ( #newline == 1 )
>>> > }
>>> > ```
>>> >
>>> > Then the issue prevails.
>>> >
>>> > Note that this requires an actually large file that contains newlines.
>>> Doing `dd if=/dev/zero bs=4M count=250 of=file.bin` and scanning that won't
>>> yield usable results.
>>> >
>>> > Den onsdag den 16. august 2017 kl. 11.43.17 UTC+2 skrev necrophcodr:
>>> > Hi Wesley,
>>> >
>>> > Sorry for the late reply, vacations and all.
>>> >
>>> > So first and foremost:
>>> >
>>> > `yara -v`
>>> > yara 3.5.0
>>> >
>>> > The files getting scanned are reporting ` internal error: 30` which
>>> I'm reading to be due to files being too large. These files are often
>>> larger than 500MB too, well above the 8MB margin.
>>> >
>>> > I've attempted to replicate it using my own instructions, coupled with
>>> your misc.yar, and the result is that it works just fine.
>>> >
>>> > So I'm guessing the issue is with my own setup, and I'll continue
>>> evaluating the specifics and return with a response when I've found the
>>> culprit.
>>> >
>>> > Den mandag den 7. august 2017 kl. 16.06.59 UTC+2 skrev Wesley Shields:
>>> > I can't replicate this behavior using 3.5.0 or latest master.
>>> >
>>> > wxs@wxs-mbp yara % cat foo
>>> > include "./global.yar"
>>> > include "./misc.yar"
>>> > wxs@wxs-mbp yara % cat global.yar
>>> > global rule fileSizeLimit { condition: filesize < 1KB }
>>> > wxs@wxs-mbp yara % cat misc.yar
>>> > rule foo { condition: true }
>>> > wxs@wxs-mbp yara % ls -l /bin/ls
>>> > -rwxr-xr-x  1 root  wheel  38624 Jul 15 00:29 /bin/ls*
>>> > wxs@wxs-mbp yara % ./yara foo /bin/ls
>>> > wxs@wxs-mbp yara %
>>> >
>>> > When you say regardless of file size are you sure you're above the
>>> 8MB? Keep in mind that 8MB is 8 * 1048576, which is 8388608.
>>> >
>>> > -- WXS
>>> >
>>> > > On Jul 28, 2017, at 7:01 AM, necrophcodr 
>>> wrote:
>>> > >
>>> > > So I've got quite a few rules, but it all comes down to this:
>>> > >
>>> > > include "./rules/global.yar"
>>> > > include "./rules/misc.yar"
>>> > >
>>> > >
>>> > > The global.yar file contains
>>> > >
>>> > > global rule fileSizeLimit { condition: filesize < 8MB }
>>> > >
>>> > >
>>> > > Any rule defined in rules/misc.yar are matched regardless of file
>>> size, but this is not what I intend. What am I doing wrong 

Re: Yara not using global rules

[necrophcodr]

Alright, I've solved the issue:

Albeit this is synthetic, running

```
for f in $(seq 0 100); do printf "\n\n\n\n\n\n\n\n\n\n" >> text.txt; 
done
```

And then

```
yara inc.yar .
```

In the directory with the yara files, yields, on my test system:

./misc.yar(9): warning: $newline is slowing down scanning (critical!)
fsL ./inc.yar
fsL ./global.yar
fsL ./misc.yar
error scanning ./text.txt: internal error: 30


And while using the newline scan is not a great idea, clearly the file is 
still being scanned in one way or another, in spite of the global rule.

Den torsdag den 17. august 2017 kl. 12.24.11 UTC+2 skrev necrophcodr:
>
> I'm afraid I cannot post the exact files. I'll create a working 
> environment that replicates all the variables required, and I'll post it 
> here when I've gotten this done.
>
> Den onsdag den 16. august 2017 kl. 16.31.35 UTC+2 skrev Wesley Shields:
>>
>> I still can not replicate your problem. 
>>
>> A couple of things to note however: 
>>
>> "internal error: 30" is because there are too many matches, which happens 
>> when a single string matches too many times. It has nothing to do with file 
>> size like you guessed. 
>>
>> Your "newline_one" rule is marked as private so it should never be 
>> reported. 
>>
>> At this point I can not replicate your problem so I'm curious if you 
>> could zip up the files you're using to do this and post them somewhere for 
>> me to see them exactly? I would need your exact YARA rules and the file you 
>> are scanning. 
>>
>> -- WXS 
>>
>> > On Aug 16, 2017, at 5:51 AM, necrophcodr  wrote: 
>> > 
>> > Alright, so I've returned with a result: 
>> > 
>> > If I have `~/inc.yar` with the following content: 
>> > 
>> > ``` 
>> > include "./global.yar" 
>> > include "./misc.yar" 
>> > ``` 
>> > 
>> > And the content of these files respectively: 
>> > 
>> > ``` 
>> > global rule fsL { condition: filesize < 8MB } 
>> > ``` 
>> > 
>> > And 
>> > 
>> > ``` 
>> > private rule newline_one { 
>> > meta: 
>> > description = "Files that contain one newline" 
>> > author = "Steffen Rytter Postas" 
>> > 
>> > strings: 
>> > $newline = "\n" 
>> > 
>> > condition: 
>> > ( #newline == 1 ) 
>> > } 
>> > ``` 
>> > 
>> > Then the issue prevails. 
>> > 
>> > Note that this requires an actually large file that contains newlines. 
>> Doing `dd if=/dev/zero bs=4M count=250 of=file.bin` and scanning that won't 
>> yield usable results. 
>> > 
>> > Den onsdag den 16. august 2017 kl. 11.43.17 UTC+2 skrev necrophcodr: 
>> > Hi Wesley, 
>> > 
>> > Sorry for the late reply, vacations and all. 
>> > 
>> > So first and foremost: 
>> > 
>> > `yara -v` 
>> > yara 3.5.0 
>> > 
>> > The files getting scanned are reporting ` internal error: 30` which I'm 
>> reading to be due to files being too large. These files are often larger 
>> than 500MB too, well above the 8MB margin. 
>> > 
>> > I've attempted to replicate it using my own instructions, coupled with 
>> your misc.yar, and the result is that it works just fine. 
>> > 
>> > So I'm guessing the issue is with my own setup, and I'll continue 
>> evaluating the specifics and return with a response when I've found the 
>> culprit. 
>> > 
>> > Den mandag den 7. august 2017 kl. 16.06.59 UTC+2 skrev Wesley Shields: 
>> > I can't replicate this behavior using 3.5.0 or latest master. 
>> > 
>> > wxs@wxs-mbp yara % cat foo 
>> > include "./global.yar" 
>> > include "./misc.yar" 
>> > wxs@wxs-mbp yara % cat global.yar 
>> > global rule fileSizeLimit { condition: filesize < 1KB } 
>> > wxs@wxs-mbp yara % cat misc.yar 
>> > rule foo { condition: true } 
>> > wxs@wxs-mbp yara % ls -l /bin/ls 
>> > -rwxr-xr-x  1 root  wheel  38624 Jul 15 00:29 /bin/ls* 
>> > wxs@wxs-mbp yara % ./yara foo /bin/ls 
>> > wxs@wxs-mbp yara % 
>> > 
>> > When you say regardless of file size are you sure you're above the 8MB? 
>> Keep in mind that 8MB is 8 * 1048576, which is 8388608. 
>> > 
>> > -- WXS 
>> > 
>> > > On Jul 28, 2017, at 7:01 AM, necrophcodr  
>> wrote: 
>> > > 
>> > > So I've got quite a few rules, but it all comes down to this: 
>> > > 
>> > > include "./rules/global.yar" 
>> > > include "./rules/misc.yar" 
>> > > 
>> > > 
>> > > The global.yar file contains 
>> > > 
>> > > global rule fileSizeLimit { condition: filesize < 8MB } 
>> > > 
>> > > 
>> > > Any rule defined in rules/misc.yar are matched regardless of file 
>> size, but this is not what I intend. What am I doing wrong here? 
>> > > 
>> > > If this is not the right place to post, that's alright, feel free to 
>> slap me on the wrist and direct me to the correct location. 
>> > > 
>> > > edit: 
>> > > 
>> > > I should mention this is using Yara 3.5.0. I don't have a chance to 
>> upgrade this within the week. 
>> > > 
>> > > 
>> > > -- 
>> > > You received this message because you are subscribed to the Google 
>> Groups "YARA" group. 
>> > > To unsubscribe