Re: Exception "Already has missing vertex on this worker"

[Yingyi Bu]

The job eventually fails because of out-of-hearbeat from tasks:

Task attempt_201309262131_0001_m_58_0 failed to report status for
602 seconds. Killing!

Any suggestions?

Yingyi



On Thu, Sep 26, 2013 at 3:27 PM, Yingyi Bu  wrote:

> I checked several other slave machines.
> Basically the map task is waiting on this trace:
>
> "main" prio=10 tid=0x098ed000 nid=0x7beb in Object.wait()
> [0x413e7000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000400108530> (a
> java.util.concurrent.ConcurrentHashMap)
> at
> org.apache.giraph.comm.netty.NettyClient.waitSomeRequests(NettyClient.java:690)
> - locked <0x000400108530> (a
> java.util.concurrent.ConcurrentHashMap)
> at
> org.apache.giraph.comm.netty.NettyClient.waitAllRequests(NettyClient.java:666)
> at
> org.apache.giraph.comm.netty.NettyWorkerClient.waitAllRequests(NettyWorkerClient.java:149)
> at
> org.apache.giraph.worker.BspServiceWorker.waitForRequestsToFinish(BspServiceWorker.java:829)
> at
> org.apache.giraph.worker.BspServiceWorker.finishSuperstep(BspServiceWorker.java:743)
> at
> org.apache.giraph.graph.GraphTaskManager.completeSuperstepAndCollectStats(GraphTaskManager.java:387)
> at
> org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:276)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
>
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Is it because I miss some setting?
>
> Yingyi
>
>
>
> On Thu, Sep 26, 2013 at 3:16 PM, Yingyi Bu  wrote:
>
>> I have 61 slave machines. Each slave machine has 16GB memory and 4 cores.
>>
>> I tried two configurations:
>> 1.   Let mapred.map.child.java.opts to be -Xmx4g, and run the job with 4
>> workers per machine on average (-w 240, try to use all the cores).
>> 2.   Let mapred.map.child.java.opts to be -Xmx16g, and run the job with 1
>> worker per machine on average (-w 60).
>>
>> I used the combiner.
>> Here are the behaviors of the two configurations:
>> 1. Configuration 1 fails with OutOfMemoryError--GC limit exceeds during
>> superstep -1.
>> 2. Configuration 2 can finish superstep -1 but hang at superstep 0 for a
>> long time (more than 40 minutes).  The status of each map task is
>> "startSuperstep: WORKER_ONLY - Attempt=0, Superstep=0".  I checked several
>> slave machines -- the CPU is not used.  Attached is the dumped stacktrace.
>> Does any one have experience with similar situations?
>>
>> Another question is: how can I effectively use all the cores in slave
>> machines?   Does each worker do multi-threading?
>> Thanks a lot!
>>
>> Yingyi
>>
>>
>>
>> On Thu, Sep 26, 2013 at 1:08 PM, Avery Ching  wrote:
>>
>>>  Hopefully you are using combiners and also re-using objects.  This can
>>> keep memory usage much lower.  Also implementing your own OutEdges can make
>>> it much more efficient.
>>>
>>> How much memory do you have?
>>>
>>> Avery
>>>
>>>
>>> On 9/26/13 12:51 PM, Yingyi Bu wrote:
>>>
>>> >> I think you may have added the same vertex 2x?
>>> I ran the job over roughly half of the graph and saw this.  However the
>>> input is not a connected components such that there might be target vertex
>>> ids which do not exist.
>>> When I ran the job over the entire graph,  I cannot see this but the job
>>> fails with exceeding GC limit (trying out-of-core now).
>>>
>>>  Yingyi
>>>
>>>
>>>
>>> On Thu, Sep 26, 2013 at 12:05 PM, Avery Ching  wrote:
>>>
  I think you may have added the same vertex 2x?  That being said, I
 don't see why the code is this way.  It should be fine.  We should file a
 JIRA.


 On 9/26/13 11:02 AM, Yingyi Bu wrote:

  Thanks, Lukas!
  I think the reason of this exception is that I run the job over part
 of the graph where some target ids do not exist.

  Yingyi


 On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec <
 lukas.naleze...@firma.seznam.cz> wrote:

>  Hi,
> Do you use partition balancing ?
>  Lukas
>
>
>
> On 09/26/13 05:16, Yingyi Bu wrote:
>
>  Hi,
>
> I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 
> machine cluster with 28GB input data.  But I got this exception:
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception 
> resolveMutations: Already has missing vertex on this worker for 20464109
>   at org.apache.giraph.gr

Re: Exception "Already has missing vertex on this worker"

[Yingyi Bu]

I checked several other slave machines.
Basically the map task is waiting on this trace:

"main" prio=10 tid=0x098ed000 nid=0x7beb in Object.wait()
[0x413e7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000400108530> (a
java.util.concurrent.ConcurrentHashMap)
at
org.apache.giraph.comm.netty.NettyClient.waitSomeRequests(NettyClient.java:690)
- locked <0x000400108530> (a
java.util.concurrent.ConcurrentHashMap)
at
org.apache.giraph.comm.netty.NettyClient.waitAllRequests(NettyClient.java:666)
at
org.apache.giraph.comm.netty.NettyWorkerClient.waitAllRequests(NettyWorkerClient.java:149)
at
org.apache.giraph.worker.BspServiceWorker.waitForRequestsToFinish(BspServiceWorker.java:829)
at
org.apache.giraph.worker.BspServiceWorker.finishSuperstep(BspServiceWorker.java:743)
at
org.apache.giraph.graph.GraphTaskManager.completeSuperstepAndCollectStats(GraphTaskManager.java:387)
at
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:276)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

Is it because I miss some setting?

Yingyi



On Thu, Sep 26, 2013 at 3:16 PM, Yingyi Bu  wrote:

> I have 61 slave machines. Each slave machine has 16GB memory and 4 cores.
>
> I tried two configurations:
> 1.   Let mapred.map.child.java.opts to be -Xmx4g, and run the job with 4
> workers per machine on average (-w 240, try to use all the cores).
> 2.   Let mapred.map.child.java.opts to be -Xmx16g, and run the job with 1
> worker per machine on average (-w 60).
>
> I used the combiner.
> Here are the behaviors of the two configurations:
> 1. Configuration 1 fails with OutOfMemoryError--GC limit exceeds during
> superstep -1.
> 2. Configuration 2 can finish superstep -1 but hang at superstep 0 for a
> long time (more than 40 minutes).  The status of each map task is
> "startSuperstep: WORKER_ONLY - Attempt=0, Superstep=0".  I checked several
> slave machines -- the CPU is not used.  Attached is the dumped stacktrace.
> Does any one have experience with similar situations?
>
> Another question is: how can I effectively use all the cores in slave
> machines?   Does each worker do multi-threading?
> Thanks a lot!
>
> Yingyi
>
>
>
> On Thu, Sep 26, 2013 at 1:08 PM, Avery Ching  wrote:
>
>>  Hopefully you are using combiners and also re-using objects.  This can
>> keep memory usage much lower.  Also implementing your own OutEdges can make
>> it much more efficient.
>>
>> How much memory do you have?
>>
>> Avery
>>
>>
>> On 9/26/13 12:51 PM, Yingyi Bu wrote:
>>
>> >> I think you may have added the same vertex 2x?
>> I ran the job over roughly half of the graph and saw this.  However the
>> input is not a connected components such that there might be target vertex
>> ids which do not exist.
>> When I ran the job over the entire graph,  I cannot see this but the job
>> fails with exceeding GC limit (trying out-of-core now).
>>
>>  Yingyi
>>
>>
>>
>> On Thu, Sep 26, 2013 at 12:05 PM, Avery Ching  wrote:
>>
>>>  I think you may have added the same vertex 2x?  That being said, I
>>> don't see why the code is this way.  It should be fine.  We should file a
>>> JIRA.
>>>
>>>
>>> On 9/26/13 11:02 AM, Yingyi Bu wrote:
>>>
>>>  Thanks, Lukas!
>>>  I think the reason of this exception is that I run the job over part of
>>> the graph where some target ids do not exist.
>>>
>>>  Yingyi
>>>
>>>
>>> On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec <
>>> lukas.naleze...@firma.seznam.cz> wrote:
>>>
  Hi,
 Do you use partition balancing ?
  Lukas



 On 09/26/13 05:16, Yingyi Bu wrote:

  Hi,

 I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 
 machine cluster with 28GB input data.  But I got this exception:

 java.lang.IllegalStateException: run: Caught an unrecoverable exception 
 resolveMutations: Already has missing vertex on this worker for 20464109
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
 org.apa

Re: Exception "Already has missing vertex on this worker"

[Avery Ching]

Hopefully you are using combiners and also re-using objects.  This can 
keep memory usage much lower. Also implementing your own OutEdges can 
make it much more efficient.


How much memory do you have?

Avery

On 9/26/13 12:51 PM, Yingyi Bu wrote:

>> I think you may have added the same vertex 2x?
I ran the job over roughly half of the graph and saw this.  However 
the input is not a connected components such that there might be 
target vertex ids which do not exist.
When I ran the job over the entire graph,  I cannot see this but the 
job fails with exceeding GC limit (trying out-of-core now).


Yingyi



On Thu, Sep 26, 2013 at 12:05 PM, Avery Ching > wrote:


I think you may have added the same vertex 2x?  That being said, I
don't see why the code is this way.  It should be fine.  We should
file a JIRA.


On 9/26/13 11:02 AM, Yingyi Bu wrote:

Thanks, Lukas!
I think the reason of this exception is that I run the job over
part of the graph where some target ids do not exist.

Yingyi


On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec
mailto:lukas.naleze...@firma.seznam.cz>> wrote:

Hi,
Do you use partition balancing ?
Lukas



On 09/26/13 05:16, Yingyi Bu wrote:

Hi,
I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 
machine cluster with 28GB input data.  But I got this exception:

java.lang.IllegalStateException: run: Caught an unrecoverable exception 
resolveMutations: Already has missing vertex on this worker for 20464109
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at 
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.IllegalStateException: resolveMutations: Already 
has missing vertex on this worker for 20464109
at 
org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
at 
org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
at 
org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
at 
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more


Does anyone know what is the possible cause of this exception?
Thanks!

Yingyi

Re: Exception "Already has missing vertex on this worker"

[Yingyi Bu]

>> I think you may have added the same vertex 2x?
I ran the job over roughly half of the graph and saw this.  However the
input is not a connected components such that there might be target vertex
ids which do not exist.
When I ran the job over the entire graph,  I cannot see this but the job
fails with exceeding GC limit (trying out-of-core now).

Yingyi



On Thu, Sep 26, 2013 at 12:05 PM, Avery Ching  wrote:

>  I think you may have added the same vertex 2x?  That being said, I don't
> see why the code is this way.  It should be fine.  We should file a JIRA.
>
>
> On 9/26/13 11:02 AM, Yingyi Bu wrote:
>
>  Thanks, Lukas!
>  I think the reason of this exception is that I run the job over part of
> the graph where some target ids do not exist.
>
>  Yingyi
>
>
> On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec <
> lukas.naleze...@firma.seznam.cz> wrote:
>
>>  Hi,
>> Do you use partition balancing ?
>>  Lukas
>>
>>
>>
>> On 09/26/13 05:16, Yingyi Bu wrote:
>>
>>  Hi,
>>
>> I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 
>> machine cluster with 28GB input data.  But I got this exception:
>>
>> java.lang.IllegalStateException: run: Caught an unrecoverable exception 
>> resolveMutations: Already has missing vertex on this worker for 20464109
>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>>  at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>  at org.apache.hadoop.mapred.Child.main(Child.java:253)
>> Caused by: java.lang.IllegalStateException: resolveMutations: Already has 
>> missing vertex on this worker for 20464109
>>  at 
>> org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
>>  at 
>> org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
>>  at 
>> org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
>>  at 
>> org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
>>  ... 7 more
>>
>>
>>
>> Does anyone know what is the possible cause of this exception?
>>
>> Thanks!
>>
>>
>> Yingyi
>>
>>
>>
>
>

Re: Exception "Already has missing vertex on this worker"

[Avery Ching]

I think you may have added the same vertex 2x?  That being said, I don't 
see why the code is this way.  It should be fine.  We should file a JIRA.


On 9/26/13 11:02 AM, Yingyi Bu wrote:

Thanks, Lukas!
I think the reason of this exception is that I run the job over part 
of the graph where some target ids do not exist.


Yingyi


On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec 
> wrote:


Hi,
Do you use partition balancing ?
Lukas



On 09/26/13 05:16, Yingyi Bu wrote:

Hi,
I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 
machine cluster with 28GB input data.  But I got this exception:

java.lang.IllegalStateException: run: Caught an unrecoverable exception 
resolveMutations: Already has missing vertex on this worker for 20464109
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.IllegalStateException: resolveMutations: Already has 
missing vertex on this worker for 20464109
at 
org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
at 
org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
at 
org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
at 
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more


Does anyone know what is the possible cause of this exception?
Thanks!

Yingyi

Re: Exception "Already has missing vertex on this worker"

[Yingyi Bu]

Thanks, Lukas!
I think the reason of this exception is that I run the job over part of the
graph where some target ids do not exist.

Yingyi


On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec <
lukas.naleze...@firma.seznam.cz> wrote:

>  Hi,
> Do you use partition balancing ?
> Lukas
>
>
>
> On 09/26/13 05:16, Yingyi Bu wrote:
>
>  Hi,
>
> I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 machine 
> cluster with 28GB input data.  But I got this exception:
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception 
> resolveMutations: Already has missing vertex on this worker for 20464109
>   at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>   at org.apache.hadoop.mapred.Child.main(Child.java:253)
> Caused by: java.lang.IllegalStateException: resolveMutations: Already has 
> missing vertex on this worker for 20464109
>   at 
> org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
>   at 
> org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
>   at 
> org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
>   at 
> org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
>   at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
>   ... 7 more
>
>
>
> Does anyone know what is the possible cause of this exception?
>
> Thanks!
>
>
> Yingyi
>
>
>

Re: Exception "Already has missing vertex on this worker"

[Lukas Nalezenec]

Hi,
Do you use partition balancing ?
Lukas


On 09/26/13 05:16, Yingyi Bu wrote:

Hi,
I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60 machine 
cluster with 28GB input data.  But I got this exception:

java.lang.IllegalStateException: run: Caught an unrecoverable exception 
resolveMutations: Already has missing vertex on this worker for 20464109
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.IllegalStateException: resolveMutations: Already has 
missing vertex on this worker for 20464109
at 
org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
at 
org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
at 
org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
at 
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more


Does anyone know what is the possible cause of this exception?
Thanks!

Yingyi

Exception "Already has missing vertex on this worker"

[Yingyi Bu]

Hi,

I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60
machine cluster with 28GB input data.  But I got this exception:

java.lang.IllegalStateException: run: Caught an unrecoverable
exception resolveMutations: Already has missing vertex on this worker
for 20464109
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.IllegalStateException: resolveMutations: Already
has missing vertex on this worker for 20464109
at 
org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
at 
org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
at 
org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
at 
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more

Does anyone know what is the possible cause of this exception?

Thanks!

Yingyi