> On March 23, 2016, 8:35 a.m., John Sirois wrote: > > I think this change stands on its own aside from the current state of the > > generated Go thrift bindings, but there has been a good deal of discussion > > about those bindings offline. Some homework below. > > > > For the case of the `TaskQuery` thrift struct, thrift 0.9.3 generates the > > following Go struct: > > ```go > > type TaskQuery struct { > > // unused field # 1 > > JobName string `thrift:"jobName,2" json:"jobName"` > > // unused field # 3 > > TaskIds map[string]bool `thrift:"taskIds,4" json:"taskIds"` > > Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses"` > > // unused field # 6 > > InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds"` > > // unused field # 8 > > Environment string `thrift:"environment,9" json:"environment"` > > SlaveHosts map[string]bool `thrift:"slaveHosts,10" json:"slaveHosts"` > > JobKeys map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys"` > > Offset int32 `thrift:"offset,12" json:"offset"` > > Limit int32 `thrift:"limit,13" json:"limit"` > > Role string `thrift:"role,14" json:"role"` > > } > > ``` > > > > This is reasonable since `api.TaskQuery{}.TaskIds == nil` is true; ie the > > collections (maps represent sets here) zero to nil. > > The issue comes in the serialization for these fields, `taskIds` is shown > > below as an example: > > ```go > > func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) { > > if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil { > > return thrift.PrependError(fmt.Sprintf("%T write field begin > > error 4:taskIds: ", p), err) > > } > > if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err != > > nil { > > return thrift.PrependError("error writing set begin: ", err) > > } > > for v, _ := range p.TaskIds { > > if err := oprot.WriteString(string(v)); err != nil { > > return thrift.PrependError(fmt.Sprintf("%T. (0) field > > write error: ", p), err) > > } > > } > > if err := oprot.WriteSetEnd(); err != nil { > > return thrift.PrependError("error writing set end: ", err) > > } > > if err := oprot.WriteFieldEnd(); err != nil { > > return thrift.PrependError(fmt.Sprintf("%T write field end > > error 4:taskIds: ", p), err) > > } > > return err > > } > > ``` > > > > So, since its safe to do so in Go (`len(p.TaskIds) == 0` and `for v, _ := > > range p.TaskIds {` loops 0 times for `p.TaskIds == nil`), the code always > > emits the `taskIds` field, whether nil or not, which presents on the other > > end of the wire as an empty set (as opposed to a null or un-set set). This > > does seem like a clear bug in the thrift compiler. > > https://issues.apache.org/jira/browse/THRIFT-3700 is similar, but on the > > deserialization side of things so I've filed > > https://issues.apache.org/jira/browse/THRIFT-3752.
Hrm, so this may not be a thrift compiler bug per-se. Marking all TaskQuery thrift fields as optional yields: ```go type TaskQuery struct { // unused field # 1 JobName *string `thrift:"jobName,2" json:"jobName,omitempty"` // unused field # 3 TaskIds map[string]bool `thrift:"taskIds,4" json:"taskIds,omitempty"` Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses,omitempty"` // unused field # 6 InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds,omitempty"` // unused field # 8 Environment *string `thrift:"environment,9" json:"environment,omitempty"` SlaveHosts map[string]bool `thrift:"slaveHosts,10" json:"slaveHosts,omitempty"` JobKeys map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys,omitempty"` Offset *int32 `thrift:"offset,12" json:"offset,omitempty"` Limit *int32 `thrift:"limit,13" json:"limit,omitempty"` Role *string `thrift:"role,14" json:"role,omitempty"` } ``` So map (set) fields are unchanged (still `nil`able), but primitives - not `nil`able before, are now represented as `nil`able pointers. This also has the effect of emitting `IsSet*` methods and respecting these methods as a gate for field serialization: ```go func (p *TaskQuery) IsSetTaskIds() bool { return p.TaskIds != nil } func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) { if p.IsSetTaskIds() { if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil { return thrift.PrependError(fmt.Sprintf("%T write field begin error 4:taskIds: ", p), err) } if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err != nil { return thrift.PrependError("error writing set begin: ", err) } for v, _ := range p.TaskIds { if err := oprot.WriteString(string(v)); err != nil { return thrift.PrependError(fmt.Sprintf("%T. (0) field write error: ", p), err) } } if err := oprot.WriteSetEnd(); err != nil { return thrift.PrependError("error writing set end: ", err) } if err := oprot.WriteFieldEnd(); err != nil { return thrift.PrependError(fmt.Sprintf("%T write field end error 4:taskIds: ", p), err) } } return err } ``` In other words, the thirft compiler for Go has a different notion of unset requiredness than the java compiler - and the perils of the global lack of specificity as to how to handle optional vs required vs <none> is well documented :/ - John ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45193/#review125024 ----------------------------------------------------------- On March 23, 2016, 10:35 a.m., John Sirois wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45193/ > ----------------------------------------------------------- > > (Updated March 23, 2016, 10:35 a.m.) > > > Review request for Aurora, David Chung, Bill Farner, and Zameer Manji. > > > Repository: aurora > > > Description > ------- > > Previously, `null` was handled differently from an empty collection in > task queries. For the Go thrift bindings, this was problematic since > zero values in Go are useful in almost all cases and in particular in the > case of maps (used to represent sets). In these cases unset `TaskQuery` > collection parameters are serialized as empty collections (empty > maps) instead of `nil` (`null`), leading to the inability to use the > query API in any natural way. > > src/main/java/org/apache/aurora/scheduler/base/JobKeys.java > | 2 +- > src/main/java/org/apache/aurora/scheduler/base/Query.java > | 2 +- > src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java > | 2 +- > src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java > | 6 ++++-- > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml > | 6 +++--- > src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java > | 20 +++++++++++++++++--- > 6 files changed, 27 insertions(+), 11 deletions(-) > > > Diffs > ----- > > src/main/java/org/apache/aurora/scheduler/base/JobKeys.java > 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 > src/main/java/org/apache/aurora/scheduler/base/Query.java > ee01eaa4d0230d6bf0909b6460f27a74f03240db > src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java > ac0bb374842741d7ccb7a83c574a90ac156af0f9 > src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java > 231a55615abfbb483667f5f8ef71d2709fc16a88 > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml > 684614ffc42dd6778c7675a6c2f81cb72c106c0e > > src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java > e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 > > Diff: https://reviews.apache.org/r/45193/diff/ > > > Testing > ------- > > NB: This change was broken out of https://reviews.apache.org/r/42756/ > since it stands on its own (although its slightly more awkward in the > mutable thrift world) and the case of the Go Aurora API client forces the > issue. > > Locally green: > ``` > ./build-support/jenkins/build.sh > ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh > ``` > > > Thanks, > > John Sirois > >