1. A method for transferring requested data over a node chain from a source node to a destination node through at least two intermediate nodes, comprising:
identifying, at a file system level, at each of the at least two intermediate nodes, prior data previously sent to or received from a subsequent node in the node chain that matches at least a portion of the requested data;
transmitting, to the subsequent node, at least one pointer to the prior data and a reconstruction recipe for reconstructing the requested data at the subsequent node based on at least the at least one pointer, when the prior data matches at least the portion of the requested data; and
transmitting, to the subsequent node, other portions of the requested data previously unsent to or un-received by the subsequent node.
2. The method of claim 1, wherein said identifying step identifies, at a non-transport layer, the prior data that matches at least the portion of the requested data.
3. The method of claim 1, wherein the file system level is at a byte level.
4. The method of claim 1, wherein the data transfer method further comprises, by at least each of the at least two destination nodes:
chunking a file at a plurality of chunking levels;
generating a respective reconstruction recipe for reconstructing the file at the subsequent node and a respective chunk archive for storing chunks for the file, at each of the plurality of chunking levels; and
transmitting the reconstruction recipe and the chunk archive corresponding to at least one of the plurality of chunking levels to the subsequent node,
wherein the file is comprised in the prior data that is used by said identifying step.
5. The method of claim 4, wherein said generating step comprises:
generating a plurality of reconstruction recipes and a plurality of chunk archives; and
selecting a given one of the plurality of reconstruction recipes and a given one of the plurality of chunk archives corresponding thereto that result in a least amount of the data being transmitted over the node chain.
6. The method of claim 4, wherein the subsequent node is a next-non-destination node in the node chain.
7. The method of claim 4, wherein the subsequent node is the destination node.
8. The method of claim 1, wherein said identifying and transmitting steps are limited to being performed by only one of the at least two intermediate nodes at each side of a slow connection, the slow connection having a data transfer rate less than a predetermined threshold.
9. The method of claim 1, further comprising maintaining a respective index and a respective database at each of the at least two intermediate nodes, the respective index identifying data stored in the respective database at a same one of the at least two intermediate nodes.
10. The method of claim 9, wherein the respective index maintained at each of the at least two intermediate nodes further identifies data stored in the respective database at other ones of the at least two intermediate nodes.
11. The method of claim 10, wherein the respective index maintained at each of the at least two intermediate nodes further identifies data stored at the source node and the destination node.
12. The method of claim 9, wherein the respective database at each of the at least two intermediate nodes is unsynchronized to the respective database at other ones of the least two intermediate nodes.
13. The method of claim 1, wherein the requested data is completely reconstructed at one of the at least two intermediate nodes, and the completely reconstructed requested data is then transmitted to a subsequent node.
14. The method of claim 1, further comprising transmitting, to the subsequent node, changes between the prior data and the requested data, when an amount of the prior data above a threshold amount matches at least the portion of the requested data.
15. The method of claim 1, wherein the method is implemented in a computer-executable program tangibly embodied on a computer readable storage medium.
16. A system for transferring requested data over a node chain from a source node to a destination node through at least two intermediate nodes, comprising:
a matching data portion identifier for identifying, at a file system level, at each of the at least two intermediate nodes, prior data previously sent to or received from a subsequent node in the node chain that matches at least a portion of the requested data; and
a transmitting device for transmitting, to the subsequent node, at least one pointer to the prior data and a reconstruction recipe for reconstructing the requested data at the subsequent node based on at least the at least one pointer, when the prior data matches at least the portion of the requested data, and for transmitting, to the subsequent node, other portions of the requested data previously unsent to or un-received by the subsequent node.
17. The system of claim 16, wherein said matching data portion identifier identifies, at a non-transport layer, the previously stored data that matches at least the portion of the requested data.
18. The system of claim 16, wherein the requested data is completely reconstructed at one of the at least two intermediate nodes, and the completely reconstructed requested data is then transmitted to a subsequent node.
19. The system of claim 16, wherein said transmitting device further transmits, to the subsequent node, changes between the prior data and the requested data, when an amount of the prior data above a threshold amount matches at least the portion of the requested data.
20. A method for transferring requested data over a node chain from a source node to a destination node through at least two intermediate nodes, comprising:
chunking a file at a plurality of chunking levels;
generating a respective reconstruction recipe for reconstructing the file at a subsequent node in the node chain and a respective chunk archive for storing chunks for the file, at each of the plurality of chunking levels;
selecting the respective reconstruction recipe and the chunk archive corresponding thereto resulting in a least amount of the data being transmitted to the subsequent node; and
transmitting the selected reconstruction recipe and the selected chunk archive to the subsequent node.
21. The method of claim 20, wherein the file is completely reconstructed at one of the at least two intermediate nodes, and the completely reconstructed file is then transmitted to a subsequent non-destination node in the node chain.
22. The method of claim 20, wherein the file is completely reconstructed at one of the at least two intermediate nodes, and the completely reconstructed file is then transmitted to the destination node.
23. A system for transferring requested data over a node chain from a source node to a destination node through at least two intermediate nodes, comprising:
a data chunker for chunking a file at a plurality of chunking levels;
a reconstruction recipe and chunk archive generator for generating a respective reconstruction recipe for reconstructing the file at a subsequent node in the node chain and a respective chunk archive for storing chunks for the file, at each of the plurality of chunking levels, and for selecting the respective reconstruction recipe and the chunk archive corresponding thereto resulting in a least amount of the data being transmitted to the subsequent node; and
a transmitting device for transmitting the selected reconstruction recipe and the selected chunk archive to the subsequent node.
24. The system of claim 23, wherein the file is completely reconstructed at one of the at least two intermediate nodes, and the completely reconstructed file is then transmitted to a subsequent non-destination node in the node chain.
25. The system of claim 23, wherein the file is completely reconstructed at one of the at least two intermediate nodes, and the completely reconstructed file is then transmitted to the destination node.
The claims below are in addition to those above.
All refrences to claim(s) which appear below refer to the numbering after this setence.
1. A method comprising:
monitoring a transmission of data packets through one of multiple ports of a server coupled to a traffic source;
characterizing the monitored transmission of data packets in accordance with a token bucket model;
generating a first parameter value characterizing fluctuations in a transmission rate of data through the port relative to a transmission rate for the monitored transmission of data packets, wherein the first parameter value is generated based on the monitoring of the transmission of data packet 5, wherein the first parameter value characterizes a size of the fluctuations relative to an average transmission rate for the monitored transmission, and wherein said generating the first parameter value comprises determining a token bucket size for the monitored transmission of data packets;
generating a second parameter value characterizing an average data transmission rate of data through the port for the monitored transmission, wherein said generating the second parameter value comprises determining a token bucket rate for the monitored transmission of data packets; and
allocating data packets in said transmission from the traffic source to at least one other port for transmission based on the first parameter value and the second parameter value, wherein the allocating of data packets is accomplished within the server.
2. The method of claim 1, further comprising allocating data packets from the traffic source for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.
3. A method of comprising:
monitoring transmissions of data packets through multiple ports of a server coupled to a traffic source;
characterizing the transmissions in accordance with a token bucket model;
generating respective first parameter values characterizing respective fluctuations in respective data transmission rates of data through respective ports relative to respective transmission rates for each of the monitored transmissions of data packets, wherein the respective first parameter values are generated based on the monitoring of the respective transmission, wherein the respective first parameter values characterize respective sizes of fluctuations in respective transmission rates of data through a respective port relative to respective average transmission rates for each of the monitored transmissions of data packets, and wherein generating the respective first parameter values comprises determining respective token bucket sizes for each of the monitored transmissions of data packets;
generating respective second parameter values characterizing respective average data transmission rates of data through a respective port for each of the monitored transmissions of data packets, wherein said generating the respective second parameter values comprises determining respective token bucket rates for each of the monitored transmissions of data packets; and
allocating outgoing data packets in said monitored transmissions from the server among the multiple ports for transmission based on the respective first parameter values and the respective second parameter values, wherein the allocating of outgoing data packets is accomplished within the server.
4. The method of claim 3, further comprising:
allocating data packets from the traffic source among the multiple ports for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.
5. A method comprising:
monitoring a transmission of data packets through one of multiple ports of a server coupled to a traffic source;
characterizing the monitored transmission of data packets in accordance with a token bucket model;
generating a first parameter value characterizing a first data transmission rate of data through the port for the monitored transmission, the first data transmission rate being averaged over a first duration of time, wherein the first parameter value is generated based on the monitoring of the transmission, and wherein said generating the first parameter value comprises determining a token bucket size for the monitored transmission of data packets;
generating a second parameter value characterizing a second data transmission rate of data through the port for the monitored transmission, the second data transmission rate being averaged over a second duration of time, the second duration of time being shorter than the first duration of time, wherein said generating the second parameter value comprises determining a token bucket rate for the monitored transmission of data packets; and
allocating outgoing data packets in said transmission from the server to at least one other port for transmission based on the first parameter value and the second parameter value, wherein said allocating the data packets is accomplished within the server.
6. The method of claim 5, further comprising allocating data packets for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.
7. An apparatus comprising:
a traffic source coupled to multiple ports of a server;
a collector configured to monitor transmissions of data packets through multiple ports, wherein the monitored transmissions of data packets are characterized in accordance with a token bucket model;
an engine configured to generate respective first parameter values characterizing respective fluctuations in respective data transmission rates of data through a port relative to respective data packets, wherein the first parameter value is generated based on the monitoring of transmissions, wherein respective first parameter values characterize respective sizes of fluctuations in respective data transmission rates of data through a respective port relative to respective average transmission rates for each of the monitored transmissions, wherein the engine is further configured to generate respective second parameter values characterizing respective average data transmission rates of data through a respective port for each of the monitored transmissions of data packets, and wherein the engine is further configured to determine respective token bucket sizes for each of the monitored transmissions of data packets and to determine respective token bucket rates for each of the monitored transmissions of data packets; and
a controller within the server, the controller configured to allocate outgoing data packets in said transmission from the server among the multiple ports for transmission based on the respective first parameter values, and the controller is further configured to allocate data packets from the traffic source among the multiple ports for transmission based on the respective first parameter values, and the respective second parameter values.
8. The apparatus of claim 7, wherein the controller is further configured to allocate data packets among the multiple ports for transmission such that the rate at which data are transmitted through each of the ports is lower than a predetermined upper bound.
9. An apparatus comprising:
a traffic source coupled to multiple ports of a server;
a collector configured to monitor a transmission of data packets through one of the multiple ports, wherein the monitored transmission of data packets is characterized in accordance with a token bucket model;
an engine configured to generate a first parameter value based on the monitoring of transmission, the first parameter value characterizing a first data transmission rate of data through the port for the monitored transmission, the first data transmission rate being averaged over a first duration of time, and to generate a second parameter value characterizing a second data transmission rate of data through the port for the monitored transmission, the second data transmission rate being averaged over a second duration of time, the second duration of time being shorter than the first duration of time, and wherein the engine is further configured to determine a token bucket size and a token bucket rate for the monitored transmission of data packets; and
a controller within the server, the controller configured to allocate outgoing data packets in said transmission from the server to at least one other port for transmission based on the first parameter value and the second parameter value.
10. The apparatus of claim 9, wherein the controller is further configured to allocate data packets from the traffic source for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.
11. An apparatus comprising:
a traffic source coupled to multiple ports of a server;
a collector configured to monitor transmissions of data packets through the multiple ports;
an engine configured to:
generate respective first parameter values based on the monitored transmission of data packets, the respective first parameter values characterizing respective first data transmission rates of data through a port for each of the monitored transmissions, the first data transmission rates being averaged over respective first durations of time, and
generate respective second parameter values characterizing respective second data transmission rates of data through a port for each of the monitored transmissions, the second data transmission rates being averaged over respective second durations of time, the respective second durations of time being shorter than the respective first durations of time; and
a controller within the server, the controller configured to allocate outgoing data packets in said transmissions from the server to at least one other port for transmission based on the respective first parameter values and the respective second parameter values;
wherein the transmissions are characterized in accordance with a token bucket model; and the engine is further configured to determine respective token bucket sizes for each of the monitored transmissions of data packets and to determine respective token bucket rates for each of the monitored transmissions of data packets.
12. The apparatus of claim 11, wherein the controller is further configured to allocate data packets for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.
13. A system comprising:
at least four ports;
a traffic source coupled with the ports of a server;
a collector configured to monitor transmissions of data packets through the ports, wherein the monitored transmissions of data packets are characterized in accordance with a token bucket model;
an engine configured to generate respective first parameter values characterizing respective fluctuations in respective data transmission rates of data through a port relative to respective transmission rates for each of the monitored transmissions of data packets, wherein the first parameter value is generated based on the monitoring of transmissions, wherein respective first parameter values characterize respective sizes of fluctuations in respective data transmission rates of data through a respective port relative to respective average transmission rates for each of the monitored transmissions, wherein the engine is further configured to generate respective second parameter values characterizing respective average data transmission rates of data through a respective port for each of the monitored transmissions of data packets, and wherein the engine is further configured to determine respective token bucket sizes for each of the monitored transmissions of data packets and to determine respective taken bucket rates for each of the monitored transmissions of data packets; and
a controller within the server, the controller configured to allocate outgoing data packets in said transmission from the server among the ports for transmission based on the respective first parameter values, and the controller is further configured to allocate data packets from the traffic source among the ports for transmission based on the respective first parameter values and the respective second parameter values.
14. The system of claim 13, wherein the controller is further configured to allocate data packets among the ports for transmission such that the rate at which data are transmitted through each of the ports is lower than a predetermined upper bound.
15. A system comprising:
at least four ports;
a traffic source coupled to the ports of a server;
a collector configured to monitor a transmission of data packets through one of the ports, wherein the monitored transmission of data packets is characterized in accordance with a token bucket model;
an engine configured to generate a first parameter value based on the monitoring of transmission, the first parameter value characterizing a first data transmission rate of data through the port for the monitored transmission, the first data transmission rate being averaged over a first duration of time, and to generate a second parameter value characterizing a second data transmission rate of data through the port for the monitored transmission, the second data transmission rate being averaged over a second duration of time, the second duration of time being shorter than the first duration of time, and wherein the engine is further configured to determine a token bucket size and a token bucket rate for the monitored transmission of data packets; and
a controller within the server, the controller configured to allocate outgoing data packets in said transmission from the server to at least one other port for transmission based on the first parameter value and the second parameter value.
16. The system of claim 15, wherein the controller is further configured to allocate data packets from the traffic source for transmission such that the rate at which data are transmitted through each of the ports is lower than a predetermined upper bound.
17. An article comprising a computer-readable medium storing instructions for causing a traffic source having multiple ports for transmitting data packets to perform operations comprising:
monitoring a transmission of data packets through one of multiple ports of a server coupled to a traffic source;
characterizing the monitored transmission of data packets in accordance with a token bucket model;
generating a first parameter value characterizing fluctuations in a transmission rate of data through the port relative to a transmission rate for the monitored transmission of data packets, wherein the first parameter value is generated based on the monitoring of the transmission of data packets, wherein the first parameter value characterizes a size of the fluctuations relative to an average transmission rate for the monitored transmission, and wherein generating the first parameter value comprises determining a token bucket size for the monitored transmission of data packets;
generating a second parameter value characterizing an average data transmission rate of data through the port for the monitored transmission, wherein generating the second parameter value comprises determining a token bucket rate for the monitored transmission of data packets; and
allocating data packets in said transmission from the traffic source to at least one other port for transmission based on the first parameter value and the second parameter value, wherein the allocating of data packets is accomplished within the server.
18. The article of claim 17, wherein allocating data packets comprises allocating data packets from the traffic source for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.
19. An article comprising a computer-readable medium storing instructions for causing a traffic source having multiple ports for transmitting data packets to perform operations comprising:
monitoring a transmission of data packets through one of multiple ports of a server coupled to a traffic source;
characterizing the monitored transmission of data packets in accordance with a token bucket model;
generating a first parameter value characterizing a first data transmission rate of data through the port for the monitored transmission, the first data transmission rate being averaged over a first duration of time, wherein the first parameter value is generated based on the monitoring of the transmission, and wherein generating the first parameter value comprises determining a token bucket size for the monitored transmission of data packets;
generating a second parameter value characterizing a second data transmission rate of data through the port for the monitored transmission, the second data transmission rate being averaged over a second duration of time, the second duration of time being shorter than the first duration of time, wherein generating the second parameter value comprise determining a token bucket rate for the monitored transmission of data packets; and
allocating outgoing data packets in said transmission from the server to at least one other port for transmission based on the first parameter value and the second parameter value, wherein allocating the data packets is accomplished within the server.
20. The article of claim 19, wherein allocating outgoing data packets comprise allocating data packets for transmission such that the rate at which data are transmitted through each of the multiple ports is lower than a predetermined upper bound.