Re: Issue 130 in memcached: Can memcache deployed on more then one server?
Updates: Status: Invalid Comment #1 on issue 130 by trond.norbye: Can memcache deployed on more then one server? http://code.google.com/p/memcached/issues/detail?id=130 Please do not file questions as bug reports. -- You received this message because you are listed in the owner or CC fields of this issue, or because you starred this issue. You may adjust your issue notification preferences at: http://code.google.com/hosting/settings -- To unsubscribe, reply using "remove me" as the subject.
Issue 130 in memcached: Can memcache deployed on more then one server?
Status: New Owner: Labels: Type-Defect Priority-Medium New issue 130 by mayongzhen: Can memcache deployed on more then one server? http://code.google.com/p/memcached/issues/detail?id=130 can memcache deployed on more then one server and the servers can communication each other ? -- You received this message because you are listed in the owner or CC fields of this issue, or because you starred this issue. You may adjust your issue notification preferences at: http://code.google.com/hosting/settings -- To unsubscribe, reply using "remove me" as the subject.
Re: A few ideas of engine framework
On 13. apr. 2010, at 17.54, KaiGai Kohei wrote:
>
> Example)
> typedef enum {
>ENGINE_FEATURE_CAS = 0, /**< has compare-and-set operation
> */
>ENGINE_FEATURE_PERSISTENT_STORAGE, /**< has persistent storage
> support */
>ENGINE_FEATURE_SECONDARY_ENGINE,/**< performs as pseudo engine */
>ENGINE_FEATURE_ACCESS_CONTROL, /**< has access control feature */
>NUM_OF_ENGINE_FEATURES,
>} ENGINE_FEATURE_CODE;
>
This is exactly what I was meaning by registering the stuff in the headed file.
I still want to be able to present this in a textual form to the user, so
that's why I want to the description there.
I know for sure that I got some ideas for thing I could want to put into a
feature code in _my_ engines, but I'm pretty sure that they are meaningless to
other folks. I don't want to inform the rest of the world about my private
extensions, so I don't want to modify a public header. At the same time I don't
want to hack my memcached server to be able to print the features in a more
sane matter...
Trond
--
To unsubscribe, reply using "remove me" as the subject.
Re: A few ideas of engine framework
(2010/04/14 1:06), Trond Norbye wrote:
>
> On 13. apr. 2010, at 01.35, KaiGai Kohei wrote:
>
>>
>>> I changed the signature for the existing "get_info" call to allow the engine
>>> to return an arbitrary number of features it support.
>>> If I remember correctly from your proposal you proposed this as a bitmask?
>>> Since this isn't going to be part of any time/space-critical loops, I think
>>> we shouldn't limit the design to a fixed number of features.
>>
>> Do you have a plan to provide guideline of the feature's signature?
>> Is it informed via feature_info->feature? or need to parse description?
>>
>
> The intention is that we define different features as constants in the
> headerfile. People may however define their own feature sets that they
> want to be able to report, but is meaningless for others to care about...
Sorry, the worth of this new method is to inform what features are supported
in this engine module, so it is completely meaningless if people defines
their own feature sets.
Could you define a set of features in the core, like error code?
And, I don't think description for each features are necessary, because it
should be obvious when we see the feature code.
Example)
typedef enum {
ENGINE_FEATURE_CAS = 0, /**< has compare-and-set operation
*/
ENGINE_FEATURE_PERSISTENT_STORAGE, /**< has persistent storage support
*/
ENGINE_FEATURE_SECONDARY_ENGINE,/**< performs as pseudo engine */
ENGINE_FEATURE_ACCESS_CONTROL, /**< has access control feature */
NUM_OF_ENGINE_FEATURES,
} ENGINE_FEATURE_CODE;
Perhaps, # of features is changed depending on initialization option.
If "use_cas=false" will be given, the default_engine does not need to
return ENGINE_FEATURE_CAS, and so on.
>>> To make it even more "flexible" I decided to pass this as an array of
>>> features,
>>> each containing a value and a description.
>>>
>>> The next thing we need to do is to start "registering" different features
>>> :-)
>>
>> In your tree, default_engine.c and corresponding files are deployed at the
>> top level directory... I think an engine module has an individual directory
>> to store corresponding files, when we register different features.
>
> The default engine is special in the way that it is bundled with memcached.
> Other engines should be developed in separate git repositories...
Hmm. It seems to me I misunderstood the meaning of the "registering" different
features.
Thanks,
--
KaiGai Kohei
--
To unsubscribe, reply using "remove me" as the subject.
diff --git a/default_engine.c b/default_engine.c
index d608cc6..e463163 100644
--- a/default_engine.c
+++ b/default_engine.c
@@ -127,6 +127,10 @@ ENGINE_ERROR_CODE create_instance(uint64_t interface,
.factor = 1.25,
.chunk_size = 48,
.item_size_max= 1024 * 1024,
+ },
+ .info = {
+ .description = "Default engine v0.1",
+ .num_features = 0
}
};
@@ -146,12 +150,9 @@ static inline hash_item* get_real_item(item* item) {
}
static const engine_info* default_get_info(ENGINE_HANDLE* handle) {
-static engine_info info = {
-.description = "Default engine v0.1",
-.num_features = 0
-};
+struct default_engine* se = get_handle(handle);
-return &info;
+return &se->info;
}
static ENGINE_ERROR_CODE default_initialize(ENGINE_HANDLE* handle,
@@ -163,6 +164,10 @@ static ENGINE_ERROR_CODE default_initialize(ENGINE_HANDLE* handle,
return ret;
}
+ /* fixup feature_info */
+ if (se->config.use_cas)
+ se->info.features[se->info.num_features++] = ENGINE_FEATURE_CAS;
+
ret = assoc_init(se);
if (ret != ENGINE_SUCCESS) {
return ret;
diff --git a/default_engine.h b/default_engine.h
index 4394569..e1d4a2f 100644
--- a/default_engine.h
+++ b/default_engine.h
@@ -120,6 +120,11 @@ struct default_engine {
struct config config;
struct engine_stats stats;
+
+ /**
+ * The feature to be supported in the default_engine
+ */
+ engine_info info;
};
char* item_get_data(const hash_item* item);
diff --git a/include/memcached/engine.h b/include/memcached/engine.h
index ae0cf9a..75b087a 100644
--- a/include/memcached/engine.h
+++ b/include/memcached/engine.h
@@ -62,6 +62,7 @@ extern "C" {
ENGINE_WANT_MORE = 0x09, /**< The engine want more data if the frontend
* have more data available. */
ENGINE_DISCONNECT = 0x0a, /**< Tell the server to disconnect this client */
+ENGINE_EACCESS = 0x0b, /**< Access control violations */
ENGINE_FAILED = 0xff /**< Generic failue. */
} ENGINE_ERROR_CODE;
@@ -327,6 +328,14 @@ extern "C" {
const char *description;
} feature_info;
+typedef enum {
+ENGINE_FEATURE_CAS = 0, /**< has compare-and-set operation */
+ENGINE_FEATURE_PERSISTENT_STORAGE, /**< has persistent storage support */
+
Re: A few ideas of engine framework
On 13. apr. 2010, at 01.17, KaiGai Kohei wrote:
>
> Rather than the server API to inform maximum number of fragments,
> it seems to me the get_item_info() should provides total number of
> fragments in the specified item, and the memcached core calls engine
> api to set up an iovector of the specified fragment of the item.
>
> If so, a part of intermediating module will be able to drop a part of
> the fragments which contains security context.
>
> Of course, a large number of iteration of append/prepend may make
> the data structure inefficient, so I think any mechanism to reorganize
> well hashed fragments...
>
I don't like the idea that the engine should be able to require the server to
support an arbitrary number of fragments. The core needs to allocate space for
the iovector for transfer of the item, so if we make this completely arbitrary
we cannot determine the memory usage we need for an item transfer.
>>> BTW, here is a point I missed it.
>>> Currently, do_add_delta() and do_store_item() with APPEND/PREPEND option
>>> assume the data field only contains the original data. So, if we inject
>>> extra attributes into the data region, it performs unexpectedly.
>>>
>>> For example, if we append {key="aaa", value="mokemoke"} after the
>>> {key="aaa", value="fugafuga"}, and both have "classified\0" as security
>>> label, it will concatenate them without eliminating extra attributes,
>>> like {key="aaa", value="fugafugaclassified\0mokemoke"}.
>>>
>>> So, we need to inform secondary module where is the data segment and where
>>> is the extra attributes of items, even if the secondary module allocate
>>> data region in flat.
>>
>> If you're storing meta-data inside the real data of an item, you would just
>> have to add your own implementation for append/prepend where you do the
>> operation on the items and then try to call a cas set operation to the
>> engine.
>
> If we can store meta-data in a certain fragment, is it unnecessary to handle
> append/preprend and delta operations in the intermediating module?
>
I really don't want to force all engines to support storing attributes (or use
io vectors)
Trond
--
To unsubscribe, reply using "remove me" as the subject.
Re: A few ideas of engine framework
On 13. apr. 2010, at 01.35, KaiGai Kohei wrote: > >> I changed the signature for the existing "get_info" call to allow the engine >> to return an arbitrary number of features it support. >> If I remember correctly from your proposal you proposed this as a bitmask? >> Since this isn't going to be part of any time/space-critical loops, I think >> we shouldn't limit the design to a fixed number of features. > > Do you have a plan to provide guideline of the feature's signature? > Is it informed via feature_info->feature? or need to parse description? > The intention is that we define different features as constants in the headerfile. People may however define their own feature sets that they want to be able to report, but is meaningless for others to care about... >> To make it even more "flexible" I decided to pass this as an array of >> features, >> each containing a value and a description. >> >> The next thing we need to do is to start "registering" different features :-) > > In your tree, default_engine.c and corresponding files are deployed at the > top level directory... I think an engine module has an individual directory > to store corresponding files, when we register different features. The default engine is special in the way that it is bundled with memcached. Other engines should be developed in separate git repositories... Cheers, Trond -- To unsubscribe, reply using "remove me" as the subject.
Re: need help
No Thanks N.SenthilRaja On 13 April 2010 12:28, Dustin wrote: > > On Apr 12, 11:32 pm, Senthil Raja wrote: > > > Will get the following error "attempting to read from closed socket", > while > > max connection is reached. > > Is there a different error you'd rather get when you exceed the > limits you've specified? > > > -- > To unsubscribe, reply using "remove me" as the subject. >
Re: A few ideas of engine framework
(2010/04/13 10:44), Trond Norbye wrote: > > On 1. apr. 2010, at 22.31, KaiGai Kohei wrote: > >> >> $ git diff origin/reworks_2 origin/reworks_3 >> -> It adds the 'features' field at engine_interface structure, to >> inform what features are supported by the loaded module. > > I couldn't see this branch anymore, but I pushed a proposal for this today. Sorry, I already removed it... > I changed the signature for the existing "get_info" call to allow the engine > to return an arbitrary number of features it support. > If I remember correctly from your proposal you proposed this as a bitmask? > Since this isn't going to be part of any time/space-critical loops, I think > we shouldn't limit the design to a fixed number of features. Do you have a plan to provide guideline of the feature's signature? Is it informed via feature_info->feature? or need to parse description? > To make it even more "flexible" I decided to pass this as an array of > features, > each containing a value and a description. > > The next thing we need to do is to start "registering" different features :-) In your tree, default_engine.c and corresponding files are deployed at the top level directory... I think an engine module has an individual directory to store corresponding files, when we register different features. Thanks, -- KaiGai Kohei -- To unsubscribe, reply using "remove me" as the subject.
Re: Am I writing to Memcached too fast?
Then, as I said, just fetch the end time. Write on MC the end time only when it changes. Let the browser do the calculation between current time and the end time. Also for a simple counter, using dinamic pages is also a viable solution. A PHP script that fetches end time from MC and does $endTime - time() calculation will be blazingly fast. -- Guille -ℬḭṩḩø- :wq -- To unsubscribe, reply using "remove me" as the subject.
Re: A few ideas of engine framework
(2010/04/13 4:28), Trond Norbye wrote:
>
> On 12. apr. 2010, at 02.18, KaiGai Kohei wrote:
>
>> Are you saying the data fragments are not necessary to be placed on the
>> continuous region in physically?
>> If so, it seems to me allocate() method needs to be revised to take an
>> arguments to indicate the number of fragments (with their lengths) on
>> allocation time?
>>
>> However, it seems to me the following description introduces all the
>> needed region is allocated on the allocation time at once, then the data
>> region will be segmented into several chunks.
>>
>
> Yes, the intention of the iovector is so that you can store the value for
> a key as multiple fragments that would be concatenated onto the wire in
> the return packet. This means that we could implement append / prepend
> really fast by just adding another element to the internal iovector of
> an item.
>
> We don't need to modify the allocate functions, because the core will call
> the method to get the iovector for the item when it needs to access the
> value.
>
> An important piece that is missing right now is the function the engine
> can call to get the maximum number of fragments that the core supports.
> I should get that in there as soon as possible.
Rather than the server API to inform maximum number of fragments,
it seems to me the get_item_info() should provides total number of
fragments in the specified item, and the memcached core calls engine
api to set up an iovector of the specified fragment of the item.
If so, a part of intermediating module will be able to drop a part of
the fragments which contains security context.
Of course, a large number of iteration of append/prepend may make
the data structure inefficient, so I think any mechanism to reorganize
well hashed fragments...
>> BTW, here is a point I missed it.
>> Currently, do_add_delta() and do_store_item() with APPEND/PREPEND option
>> assume the data field only contains the original data. So, if we inject
>> extra attributes into the data region, it performs unexpectedly.
>>
>> For example, if we append {key="aaa", value="mokemoke"} after the
>> {key="aaa", value="fugafuga"}, and both have "classified\0" as security
>> label, it will concatenate them without eliminating extra attributes,
>> like {key="aaa", value="fugafugaclassified\0mokemoke"}.
>>
>> So, we need to inform secondary module where is the data segment and where
>> is the extra attributes of items, even if the secondary module allocate
>> data region in flat.
>
> If you're storing meta-data inside the real data of an item, you would just
> have to add your own implementation for append/prepend where you do the
> operation on the items and then try to call a cas set operation to the
> engine.
If we can store meta-data in a certain fragment, is it unnecessary to handle
append/preprend and delta operations in the intermediating module?
Thanks,
--
KaiGai Kohei
--
To unsubscribe, reply using "remove me" as the subject.
