1. One or more computer-storage media having computer-executable instructions embodied thereon that, when executed, perform a method for generating distributed dataflow graphs, the method comprising:
receiving a plurality of data elements;
reading and parsing the plurality of data elements to form a sequential dataflow graph having a plurality of vertices;
applying at least one heuristic to the sequential dataflow graph;
determining at least one of the plurality of vertices that is capable of being performed multiple times in parallel;
dividing the sequential dataflow graph into a plurality of subgraphs based upon the determined at least one vertex that is capable of being performed multiple times in parallel;
replicating the plurality of subgraphs that are determined to be capable of being performed multiple times in parallel, wherein a respective number of replications of each of the plurality of subgraphs depends on at least one of available resources and external constraints; and
generating a distributed dataflow graph by connecting the replicated subgraphs.
2. The one or more computer-storage media of claim 1, wherein the sequential dataflow graph is a direct acyclic graph.
3. The one or more computer-storage media of claim 1, further comprising processing the plurality of data elements in accordance with the distributed dataflow graph.
4. The one or more computer-storage media of claim 1, wherein the sequential dataflow graph is comprised of the plurality of vertices and a plurality of edges, wherein each of the plurality of vertices represents a data transformation step, and wherein each of the plurality of edges represents the flow of data elements between the vertices.
5. The one or more computer-storage media of claim 2, wherein each of the data transformation steps includes at least one of a filtering step, a looking-up step, an aggregating step, a joining step, a merging step, a unioning step, an auto-partitioning step, a topbottom step, and a sorting step.
6. The one or more computer-storage media of claim 1, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one available resource, and wherein the at least one available resource comprises a quantity of processors that is capable of performing at least one of the plurality of vertices.
7. The one or more computer-storage media of claim 1, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one external constraint, and wherein the at least one external constraint comprises a layout of the data elements.
8. The one or more computer-storage media of claim 1, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one external constraint, and wherein the at least one external constraint comprises a quantity of the plurality of data elements entering each of the plurality of vertices.
9. The one or more computer-storage media of claim 1, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one external constraint, and wherein the at least one external constraint comprises a user input.
10. The one or more computer-storage media of claim 1, wherein generating a distributed dataflow graph by connecting the replicated subgraphs is based upon one or more semantics of each vertex.
11. One or more computer storage media having computer-executable instructions embodied thereon that, when executed, generate distributed dataflow graphs, the system comprising:
a receiving component that receives a plurality of data elements;
a reading and parsing component that reads and parses the plurality of data elements to form a sequential dataflow graph having a plurality of vertices;
an application component that applies at least one heuristic to the sequential dataflow graph to determine at least one of the plurality of vertices that is capable of being performed multiple times in parallel;
a dividing component that divides the sequential dataflow graph into a plurality of subgraphs based upon the determined at least one vertex that is capable of being performed multiple times in parallel;
a replicating component that replicates the plurality of subgraphs that are determined to be capable of being performed multiple times in parallel, wherein a respective number of replications of each of the plurality of subgraphs depends on at least one of available resources and external constraints; and
a generation component that generates a distributed dataflow graph by connecting the replicated subgraphs.
12. The one or more computer-storage media of claim 11, further comprising processing the plurality of data elements in accordance with the distributed dataflow graph.
13. The one or more computer-storage media of claim 11, wherein the sequential dataflow graph is comprised of the plurality of vertices and a plurality of edges, wherein each of the plurality of vertices represents a data transformation step, and wherein each of the plurality of edges represents the flow of data elements between the vertices.
14. The one or more computer-storage media of claim 12, wherein each of the data transformation steps include at least one of a filtering step, a looking-up step, an aggregating step, a joining step, a merging step, a unioning step, an auto-partitioning step, a topbottom step, and a sorting step.
15. The one or more computer-storage media of claim 11, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one available resource, and wherein the at least one available resource comprises a quantity of processors that is capable of performing at least one of the plurality of vertices.
16. The one or more computer-storage media of claim 11, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one external constraint, and wherein the at least one external constraint comprises the layout of the data elements.
17. The one or more computer-storage media of claim 11, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one external constraint, and wherein the at least one external constraint comprises a quantity of the plurality of data elements entering each of the plurality of vertices.
18. The one or more computer-storage media of claim 11, wherein the respective number of replications of each of the plurality of subgraphs depends on at least one external constraint, and wherein the at least one external constraint comprises a user input.
19. The one or more computer-storage media of claim 11, wherein generating a distributed dataflow graph by connecting the replicated subgraphs is based upon one or more semantics of each vertex.
20. A computerized method for processing a plurality of data elements using a distributed dataflow graph, the method comprising:
receiving a plurality of data elements;
reading and parsing the plurality of data elements to form a direct acyclic sequential dataflow graph, wherein the direct acyclic sequential dataflow graph is comprised of a plurality of vertices and a plurality of edges, and wherein each of the plurality of vertices represents a data transformation step, and wherein each of the plurality of edges represents the flow of data elements between the vertices;
applying at least one heuristic to the direct acyclic sequential dataflow graph;
determining at least one of the plurality of vertices that is capable of being performed multiple times in parallel;
generating a distributed execution plan based upon the determined at least one vertex that is capable of being performed multiple times in parallel;
dividing the sequential dataflow graph into a plurality of subgraphs based upon the distributed execution plan;
replicating at least one of the plurality of subgraphs, wherein a respective number of replications of each of the plurality of subgraphs depends on at least one of a quantity of processors that is capable of performing at least one of the plurality of vertices, the layout of the data elements, the quantity of the plurality of data elements entering each of the plurality of vertices, and a user input;
generating a distributed dataflow graph by connecting the replicated subgraphs wherein connecting the replicated subgraphs is based upon a plurality of semantics of each vertex; and
processing the plurality of data elements in accordance with the distributed dataflow graph.
The claims below are in addition to those above.
All refrences to claim(s) which appear below refer to the numbering after this setence.
1. A method for managing an information transmission line, comprising:
monitoring an information transmission line;
detecting a disturbance on the information transmission line;
displaying the disturbance as a graphical representation;
comparing the disturbance to a preset threshold; and
triggering an alert if the disturbance is greater than the preset threshold or the number of disturbances less than the preset threshold meets a preset number within a preset time period.
2. The method of claim 1, wherein the disturbance is selected from the group consisting of a vibration, a frequency change, an acoustic change, and a change in distance based on reflectometer reading.
3. The method of claim 1, wherein the alert comprises an email, a phone text, a sound, or a phone call.
4. The method of claim 1, further comprising initiating a response.
5. The method of claim 4, wherein the response is selected from the group consisting of disabling a data collection, adjusting a physical security device, re-routing a data collection, performing a network analytic, notifying a personnel, and combinations thereof
6. The method of claim 1, further comprising evaluating a characteristic of the disturbance.
7. The method of claim 6, wherein the characteristic is the location of the disturbance or the type of disturbance.
8. The method of claim 7, further comprising initiating a response if the characteristic of the disturbance meet a predefined standard.
9. The method of claim 1, further comprising automatically discovering and enrolling a monitoring device.
10. The method of claim 1, further comprising collecting data on the disturbance.
11. The method of claim 1, wherein the step of monitoring the information transmission line comprises monitoring a segment of the information transmission line.
12. The method of claim 1, wherein the information transmission line comprises a trunk cable.
13. A computer-readable medium for storing computer instructions therein, the computer-readable medium comprising a set of instructions which when executed causes a processor to perform a method for managing an information transmission line, the method comprising:
monitoring an information transmission line;
detecting a disturbance on the information transmission line;
displaying the disturbance as a graphical representation;
comparing the disturbance to a preset threshold; and
triggering an alert if the disturbance is greater than the preset threshold or the number of disturbances less than the preset threshold meets a preset number within a preset time period.
14. The computer-readable medium of claim 13, wherein the method further comprises initiating a response.
15. The computer-readable medium of claim 13, wherein the method further comprises evaluating a characteristic of the disturbance.
16. The computer-readable medium of claim 13, wherein the method further comprises automatically discovering and enrolling a monitoring device.
17. The computer-readable medium of claim 13, wherein the method further comprises collecting data on the disturbance.
18. A system for managing an information transmission line, comprising:
a computer comprising a set of instructions which when executed causes a processor to perform a method for managing an information transmission line, the method comprising:
monitoring an information transmission line;
detecting a disturbance on the information transmission line;
displaying the disturbance as a graphical representation;
comparing the disturbance to a preset threshold; and
triggering an alert if the disturbance is greater than the preset threshold or the number of disturbances less than the preset threshold meets a preset number within a preset time period;
an intrusion detector;
an optical line terminal or network switch;
an optical circuit switch;
an optical test access point device; and
a network analytic tool.
19. The system of claim 18, wherein the network analytic tool is selected from the group consisting of network monitor, security monitor, network recorder, and network analyzer.
20. The system of claim 18, further comprising a video camera.