[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171320#comment-14171320
]
Noble Paul edited comment on SOLR-6476 at 10/14/14 6:28 PM:
actually both format is supported.I will ensure that the documentation carries
both examples
But it is useful when order is important.
For the /update path it was a not a great choice because they are usually
written down by libraries which don't know how to write it . schema is mostly
written by hand. and humans are fine writing the repeated stuff if it helps
readability
was (Author: noble.paul):
actually both format is supported
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: New Feature
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Fix For: 5.0, Trunk
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171320#comment-14171320
]
Noble Paul edited comment on SOLR-6476 at 10/14/14 6:29 PM:
actually both formats are supported.I will ensure that the documentation
carries both examples
But it is useful when order is important.
For the /update path it was a not a great choice because they are usually
written down by libraries which don't know how to write it . schema is mostly
written by hand. and humans are fine writing the repeated stuff if it helps
readability
was (Author: noble.paul):
actually both format is supported.I will ensure that the documentation carries
both examples
But it is useful when order is important.
For the /update path it was a not a great choice because they are usually
written down by libraries which don't know how to write it . schema is mostly
written by hand. and humans are fine writing the repeated stuff if it helps
readability
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: New Feature
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Fix For: 5.0, Trunk
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154985#comment-14154985
]
Noble Paul edited comment on SOLR-6476 at 10/1/14 4:28 PM:
---
bq.Aren't you going to backport this to branch_5x?
yes but I am waiting for [~thelabdude] to port the changes from SOLR-6249 tobe
ported first
was (Author: noble.paul):
bq.Aren't you going to backport this to branch_5x?
yes
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: New Feature
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Fix For: 5.0, Trunk
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153438#comment-14153438
]
Steve Rowe edited comment on SOLR-6476 at 9/30/14 5:50 PM:
---
Thanks for the changes, Noble, those look fine - two new issues:
(*edit*: {{System.getCurrentTimeMillis()}} - {{currentTimeMillis()}})
# Didn't [~markrmil...@gmail.com] convert all timing stuff in Solr to use
{{nanoTime()}} instead of {{currentTimeMillis()}}? If so, shouldn't we use
{{nanoTime()}} here too? (This applies to [~thelabdude]'s SOLR-6249 work as
well.)
# In {{SchemaManager.waitForOtherReplicasToUpdate()}}, called from
{{doOperations()}}, you send {{-1}} in as {{maxWaitSecs}} to
{{ManagedIndexSchema.waitForSchemaZkVersionAgreement()}} when the timeout has
been exceeded, but AFAICT negative values aren't handled appropriately there,
e.g. it gets sent in unexamined to {{ExecutorService.invokeAll()}}:
{code:java}
private ListString doOperations(ListOperation operations){
int timeout = req.getParams().getInt(BaseSolrResource.UPDATE_TIMEOUT_SECS,
-1);
long startTime = System.currentTimeMillis();
[...]
managedIndexSchema.persistManagedSchema(false);
core.setLatestSchema(managedIndexSchema);
waitForOtherReplicasToUpdate(timeout, startTime);
[...]
}
private void waitForOtherReplicasToUpdate(int timeout, long startTime) {
if(timeout 0 [...]){
[...]
ManagedIndexSchema.waitForSchemaZkVersionAgreement([...],
getTimeLeftInSecs(timeout, startTime));
}
}
}
private int getTimeLeftInSecs(int timeout, long startTime) {
long timeLeftSecs = timeout - ((System.currentTimeMillis() - startTime)
/1000);
return (int) (timeLeftSecs 0 ?timeLeftSecs: -1);
}
{code}
was (Author: steve_rowe):
Thanks for the changes, Noble, those look fine - two new issues:
# Didn't [~markrmil...@gmail.com] convert all timing stuff in Solr to use
{{nanoTime()}} instead of {{System.getCurrentTimeMillis()}}? If so, shouldn't
we use {{nanoTime()}} here too? (This applies to [~thelabdude]'s SOLR-6249
work as well.)
# In {{SchemaManager.waitForOtherReplicasToUpdate()}}, called from
{{doOperations()}}, you send {{-1}} in as {{maxWaitSecs}} to
{{ManagedIndexSchema.waitForSchemaZkVersionAgreement()}} when the timeout has
been exceeded, but AFAICT negative values aren't handled appropriately there,
e.g. it gets sent in unexamined to {{ExecutorService.invokeAll()}}:
{code:java}
private ListString doOperations(ListOperation operations){
int timeout = req.getParams().getInt(BaseSolrResource.UPDATE_TIMEOUT_SECS,
-1);
long startTime = System.currentTimeMillis();
[...]
managedIndexSchema.persistManagedSchema(false);
core.setLatestSchema(managedIndexSchema);
waitForOtherReplicasToUpdate(timeout, startTime);
[...]
}
private void waitForOtherReplicasToUpdate(int timeout, long startTime) {
if(timeout 0 [...]){
[...]
ManagedIndexSchema.waitForSchemaZkVersionAgreement([...],
getTimeLeftInSecs(timeout, startTime));
}
}
}
private int getTimeLeftInSecs(int timeout, long startTime) {
long timeLeftSecs = timeout - ((System.currentTimeMillis() - startTime)
/1000);
return (int) (timeLeftSecs 0 ?timeLeftSecs: -1);
}
{code}
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152063#comment-14152063
]
Noble Paul edited comment on SOLR-6476 at 9/29/14 6:55 PM:
---
bq.AFAICT, a bulk Schema API request with a new field using a new field type in
the same request can fail depending on the order of the specified operations
Yes it fails. Works as designed. This is exactly the same behavior you will see
in an RDBMS as well. Order is important
was (Author: noble.paul):
bq.AFAICT, a bulk Schema API request with a new field using a new field type in
the same request can fail depending on the order of the specified operations
Yes it fails. Works as designed. This is exactly the same behavior you will see
in an RDBMS as well
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147935#comment-14147935
]
Noble Paul edited comment on SOLR-6476 at 9/26/14 11:22 AM:
bq.It can, but you don't have any distributed tests at all.
I should fix that. May be separately
bq.You removed some sugar methods,
Yeah, the class was getting very hard to read with all the sugar methods and I
prefer them removed
bq.You misunderstand me: SolrTestCaseJ4.json(String)
ok , I got it . I'll fix it
I'll fix the javadocs , in the next patch.
bq.It keeps retrying when another schema modification request gets in first.
Please fix this - you didn't change anything at all here. There is no way we
should be giving up here just because other modifications are happening.
I disagree with this idea. It is not necessary that every command has to
succeed. Whatever sucseeds , will have to do so with the given time/resources.
Otherwise , it should just fail
was (Author: noble.paul):
bq.It can, but you don't have any distributed tests at all.
I should fix that. May be separately
bq.You removed some sugar methods,
Yeah, the class was getting very hard to read with all the sugar methods and I
prefer them removed
bq.You misunderstand me: SolrTestCaseJ4.json(String)
ok , I got it . I'll fix it
I'll fix the javadocs , in the next patch.
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144498#comment-14144498
]
Noble Paul edited comment on SOLR-6476 at 9/23/14 9:47 AM:
---
bq.The bulk schema api should be added to TestCloudManagedSchemaConcurrent
Why can't it be in a new class?
bq.TestBulkSchemaAPI has lots of backslash-double-quotes in JSON strings
That was deliberate. Most JSON writers use double quotes.
bq.In SchemaManager.doOperation(), you limit retries to MAX_TRIES
I can make the variable bigger , but I somehow hate infinite loops
bq.SchemaManager.ADD_FIELD_TYPE should be add-field-type
sure
bq.SchemaManager has lots of non-generic collections
will fix them
bq.IndexSchema.addDynamicFields() is missing the persist param you added to all
the other add*() methods;
will fix
bq.Most of the add*() javadocs in IndexSchema say that persistence always
happens, but it doesn't if persist=false
We need to remove those methods w/ persist option
was (Author: noble.paul):
bq.The bulk schema api should be added to TestCloudManagedSchemaConcurrent
Why can't it be in a new class?
bq.TestBulkSchemaAPI has lots of backslash-double-quotes in JSON strings
That was deliberate. Most JSON writers use double quotes.
bq.In SchemaManager.doOperation(), you limit retries to MAX_TRIES
I can make the variable bigger , but I somehow hate infinite loops
bq.SchemaManager.ADD_FIELD_TYPE should be add-field-type
sure
bq.SchemaManager has lots of non-generic collections
will fix them
bq.IndexSchema.addDynamicFields() is missing the persist param you added to all
the other add*() methods;
will fix
bq.Most of the add*() javadocs in IndexSchema say that persistence always
happens, but it doesn't if persist=false
will fix
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144498#comment-14144498
]
Noble Paul edited comment on SOLR-6476 at 9/23/14 9:47 AM:
---
bq.The bulk schema api should be added to TestCloudManagedSchemaConcurrent
Why can't it be in a new class?
bq.TestBulkSchemaAPI has lots of backslash-double-quotes in JSON strings
That was deliberate. Most JSON writers use double quotes.
bq.In SchemaManager.doOperation(), you limit retries to MAX_TRIES
I can make the variable bigger , but I somehow hate infinite loops
bq.SchemaManager.ADD_FIELD_TYPE should be add-field-type
sure
bq.SchemaManager has lots of non-generic collections
will fix them
bq.IndexSchema.addDynamicFields() is missing the persist param you added to all
the other add*() methods;
will fix
bq.Most of the add*() javadocs in IndexSchema say that persistence always
happens, but it doesn't if persist=false
We need to remove those methods without persist option
was (Author: noble.paul):
bq.The bulk schema api should be added to TestCloudManagedSchemaConcurrent
Why can't it be in a new class?
bq.TestBulkSchemaAPI has lots of backslash-double-quotes in JSON strings
That was deliberate. Most JSON writers use double quotes.
bq.In SchemaManager.doOperation(), you limit retries to MAX_TRIES
I can make the variable bigger , but I somehow hate infinite loops
bq.SchemaManager.ADD_FIELD_TYPE should be add-field-type
sure
bq.SchemaManager has lots of non-generic collections
will fix them
bq.IndexSchema.addDynamicFields() is missing the persist param you added to all
the other add*() methods;
will fix
bq.Most of the add*() javadocs in IndexSchema say that persistence always
happens, but it doesn't if persist=false
We need to remove those methods w/ persist option
Create a bulk mode for schema API
-
Key: SOLR-6476
URL: https://issues.apache.org/jira/browse/SOLR-6476
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
Labels: managedResource
Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
SOLR-6476.patch
The current schema API does one operation at a time and the normal usecase is
that users add multiple fields/fieldtypes/copyFields etc in one shot.
example
{code:javascript}
curl http://localhost:8983/solr/collection1/schema -H
'Content-type:application/json' -d '{
add-field: {
name:sell-by,
type:tdate,
stored:true
},
add-field:{
name:catchall,
type:text_general,
stored:false
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
