Difference between revisions of "Socio-Technical Features of Systems of Systems"

From SEBoK
Jump to: navigation, search
(updated INCOSE Vision 2025 reference)
Line 2: Line 2:
 
Most [[System of Systems (SoS) (glossary)|systems of systems]] (SoS) are socio-technical systems that are composed of a number of interdependent resources, such as, people, processes, information, and technology that must interact with each other and their [[Environment (glossary)|environment]] in support of a common [[mission (glossary)]] (See also [[Enterprise Systems Engineering]]).
 
Most [[System of Systems (SoS) (glossary)|systems of systems]] (SoS) are socio-technical systems that are composed of a number of interdependent resources, such as, people, processes, information, and technology that must interact with each other and their [[Environment (glossary)|environment]] in support of a common [[mission (glossary)]] (See also [[Enterprise Systems Engineering]]).
  
==Socio-Technical Aspects of Systems of Systems Engineering==
+
==Managing the Socio-Technical Features of Systems of Systems==
Engineering a systems of systems often entails more than simply integrating physical systems but also incorporates integration of the people and processes associated with the constituent systems.  In most SoS cases, each constituent system has its own users and stakeholders with their own operating processes, objectives, motivations and constraints as well as its own technical development processes, funding mechanisms, and lifecycle approachesThis affects systems engineering for SoS in several ways.
+
The managerial and operational independence of constituent systems of a SoS (Maier, 1998) imply that in general, SoS are socio-technical systems.  Turning to views outside of Systems Engineering, Ergonomists regard socio-technical systems as having the following characteristics (Maguire, 2014):
 +
 
 +
* There are collective operational tasks,
 +
* They contain social and technical sub-systems,
 +
* They are open systems (i.e. strongly interacting with their environments), and 
 +
* The concept of the system being an unfinished system. 
 +
 
 +
These are also characteristics of Systems of Systems.  Klein (2014) has noted that approaches to socio-technical systems can take the two perspectives of “system affects people” or “people affect system”, depending upon how the system boundary is drawn.  It is generally true for systems that consideration of their context requires socio-technical aspects to be taken into account.
 +
 
 +
Although there are many matters concerning the socio-technical aspects of SoS, there are two main issues, which are somewhat related, requiring attention.  The first is the need for appropriate governance structures, given that operational and/or managerial independence affects top-down direction of the SoS and may compromise achievement of the SoS goal(s).  The second issue is a lack of situational awareness of managers, operators, or other stakeholders of the SoS, so that they may not understand the impact of their local decisions on the wider SoS.
 +
 
 +
==SoS Governance==
 +
Generally, design and operation of complex systems is concerned with control, but the classification of SoS (Dahmann, et. al., 2008) is based on the notion of diminishing central control, as the types go from directed to virtual.  Sauser, et. al. (2009) has described the ‘control paradox of SoS’ and asserted that for SoS, ‘management’ is replaced by ‘governance’.  Thus in C2 terms, the emphasis in SoS is on command, rather than control: ‘Control is a function of rules, time, and bandwidth; whereas command is a function of trusts, influence, fidelity, and agility’.  Trusts, influence, and fidelity are intrinsically human qualities.  The matters relating to the organizational aspects, such as these, are relevant to Enterprise Systems . Siemieniuch and Sinclair (2014) highlighted the complexity issues with respect to SoS by drawing attention to a variety of organizational characteristics that concern the interactions among system operators in a SoS:
 +
 
 +
Many agents, of different kinds
 +
Some degree of behavioral autonomy for agents
 +
Multiple steady states for agents
 +
Interactions between agents in an environment
 +
Lots of connections between agents
 +
Agents communicating in parallel
 +
Effects of an evolving environment
 +
Effects of evolving agents
 +
Interactions between different goals within an agent
 +
Interactions between agents with different goals
 +
Language/culture differences between agents
 +
 
 +
A major governance issue for SoS is understanding the ownership of, and making reliable estimates of risk (Fovino & Masera, 2007).  High levels of connectivity, and the potential for emergent behavior due to the interactions of separately owned/operated constituent systems, means that significant risks may go unacknowledged and their mitigations unplanned.
 +
 
 +
In general, governance can be summed up by asking three connected questions (Siemieniuch and Sinclair, 2014):
 +
* Are we doing the right things (leadership)?
 +
* Are we doing those things right (management)?
 +
* How do we know this (metrics and measurements)?
 +
 
 +
Currently, there is no accepted framework for addressing these questions in a SoS context, but Henshaw et. al. (2013) have highlighted architectures as an important means through which governance may be clarifiedThey postulate that a SoS can be regarded as a set of trust and contract relationships between systems (i.e. including both informal and formal relationships).  The systems architect of a constituent system must, therefore, address trust issues for each participating organization in the overall enterprise with which his/her system must interoperate.  For SoS, technical engineering governance is concerned with defining and ensuring compliance with trust at the interface between constituent systems.  An example of difficulty managing the interfaces in a SoS is provided in the Cassini-Huygens mission case study .
 +
 
 +
==Situational Awareness==
 +
Situational awareness is a decision maker’s understanding of the environment in which he/she takes a decision; it concerns information, awareness, perception, and cognition.  Endsley (1995) emphasizes that situational awareness is a state of knowledge.  There are numerous examples of SoS failure due to the operator of one constituent system making decisions based on inadequate knowledge of the overall SoS (big picture).  Fratricide incidents are among the most shocking (see for example Rasmussen, 2007).
 +
 
 +
On the other hand, SoS development is also viewed as the means through which improved situational awareness may be achieved (Van der Laar, et. al., 2013).  In the defense environment, Network Enabled Capability (NEC) was a system of systems approach motivated by the objective of making better use of information sharing to achieve military objectives.  Court (2005) identified that the NEC benefits chain was concerned with the quality of decision making, which relies on the quality of shared awareness, which relies, in turn, on networks and information quality; a crucial element is the sharing of information, knowledge, and understanding.  NEC is predicated on the ability to share useful information effectively among the stakeholders that need it.  It is concluded that improving situational awareness will improve SoS performance, or at least reduce the risk of failures at the SoS levelThus, the principles which govern the organization of the SoS should support sharing information effectively across the network; in essence, ensuring that every level of the interoperability spectrum is adequately serviced.  Operators need insight into the effect that their own local decisions may have on the changing SoS or environment; similarly they need to understand how external changes will affect the systems that they own.
  
First, this means that the SoS systems engineer needs to consider the operating processes of the constituent systems and how these will affect the systems of systems.  This includes considering changes in the operations of systems to meet the needs of the SoS and how these will affect the constituent systems which often continue to support their original users concurrently with the SoS.  In effect in many SoS, SoSE includes engineering operational and social processes as well as the technical systems. Examples of emerging ‘soft’ issues that are critical to the design and operation of systems of systems include issues involving autonomy, authority, responsibility and ethics, impact of culture and cultural attributes on multinational and multicultural team performance, system of systems ethics, governance, and regulation, and shared/distributed situational awareness (Hubbard et al. 2010).
 
  
Second, this means that the SoS systems engineering needs to consider the development processes of the systems including their current state of development (e.g. in development, fielded, evolving) and how this affects their ability to change to meet SoS needs.  This can place constraints on the architecture for the SoS as is discussed in the next section.  It also can introduce complexity into SoS development since it is often the case that different constituent systems may be on different development schedules making it difficult to synchronize changes across the systems in an SoS.  This can lead to challenges in SoS verification, validation and testing (REF) as well as in maintaining operational capability in the face of asynchronous changes in systems where there are interdependencies among the systems. This can be further complicated when the life cycle approaches of the constituent systems differ (Boehm and Turner, 2004).
 
  
Finally, as noted above, many SoS are in effect socio-technical systems. Socio-technical systems are composed of a number of interdependent resources, such as, people, processes, information, and technology that must interact with each other and their environment in support of a common mission.  Increasingly, (INCOSE 2014) systems engineering views this class of system as offering an opportunity for a broadened contribution of systems engineering approaches.  Since a socio-technical systems are comprised of multiple, independent systems which together provide a new capability, they can be viewed from an SoS perspective.  These socio-technical systems are sometimes referred to as enterprise systems (Rhodes et al. 2009). The relationship between SoS and Enterprise SE approaches to socio-technical systems is an active topic of discussion.
 
  
==Dealing with Socio-Technical Issues in an SoS==
 
Many of the issues associated with ‘soft’ or organizational aspects of an SoS often exhibit many of the characteristics of so-called wicked problem (Rittel and Webber 1973), including:
 
* problems are extremely complex and not bounded or stable,
 
* they do not have uniquely correct solutions, but rather solutions that are either better or worse than others, and they also do not have a definitive formulation,
 
* SoS requirements are often volatile with changing constraints and moving targets,
 
* stakeholders have different views, and
 
* understanding the whole context is challenging but critical.
 
  
These issues relate to both hard (mechanical, electronic, and [[software (glossary)|software]]) and soft (people, organizations, and regulatory) systems considerations. Research must include mixed methods and approaches (Conklin 2005) that include both quantitative and qualitative techniques, which makes this a very challenging area intellectually.
 
  
 
==References==  
 
==References==  
Line 25: Line 52:
 
===Works Cited===
 
===Works Cited===
  
Bernus, P., L. Nemes, and G. Schmidt. 2003. ''Handbook on Enterprise Architecture.'' Heidelberg, Germany: Springer-Verlag.
+
Court, G. (2005) Validating the NEC benefits chain in 11th ICCRTS. Cambridge, UK. Retrieved from http://www.dodccrp.org/events/11th_ICCRTS/html/papers/155.pdf
 +
 
 +
Dahmann, J. S. & Baldwin, K. J. (2008) Understanding the Current State of US Defense Systems of Systems and the Implications for Systems Engineering, 2nd Annual IEEE Systems Conference, 1–7. http://doi.org/10.1109/SYSTEMS.2008.4518994
 +
 
 +
Endsley, M. R. (1995) Toward a Theory of Situation Awareness in Dynamic Systems, J. Human Factors and Ergonomics Soc., 37(1), 32–64. http://doi.org/10.1518/001872095779049543
 +
 
 +
Fovino, I. N., & Masera, M. (2007) Emergent disservices in interdependent systems and system-of-systems, in Proc. IEEE International Conference on Systems, Man and Cybernetics, Vol. 1, pp. 590–595. http://doi.org/10.1109/ICSMC.2006.384449
  
Castka, P.B. 2001. "Factors Affecting the Successful Implementation of High Performance Teams." ''Team Performance Management.'' 7 (7/8): 123-134.  
+
Henshaw, M. J. de C., Siemieniuch, C. E., & Sinclair, M. A. (2013) Technical and Engineering Governance in the Context of Systems of Systems, in NATO SCI Symp. Architecture Assessment for NEC (pp. 1–10). Tallinn, Es. NATO STO.
  
Chena, D., G. Doumeingtsb, F. Vernadatc. 2008. "Architectures for enterprise integration and interoperability: Past, present and future." ''Computers in Industry.'' 59 (7): 647-659.
+
Henshaw, M. (2014) A Socio-Technical Perspective on SoSE, in Lecture Series in Systems of Systems Engineering for NATO Defence Applications (SCI-276). NATO CSO.
  
Curtis, B., W.E. Hefley, and S.A. Miller. 2009. ''People Capability Maturity Model (P-CMM)'', version 2.0, 2nd ed. Pittsburgh, PA, USA: Software Engineering Institute, Carnegie Mellon University. Available: http://repository.cmu.edu/cgi/viewcontent.cgi?article=1048&context=sei.
+
Klein, L. (2014) What do we actually mean by ‘sociotechnical’? On values, boundaries and the problems of language, Appl. Ergon., vol. 45, no. 2 PA, pp. 137–142.
  
Conklin, J. 2005. ''Dialogue Mapping: Building Shared Understanding of Wicked Problems,'' 1st ed. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd.
+
Maguire, M. (2014) Socio-technical systems and interaction design - 21st century relevance, Appl. Ergon., vol. 45, no. 2 PA, pp. 162–170.
  
Hubbard, E-M., C.E. Siemieniuch, M.A. Sinclair, and A. Hodgson. 2010. "Working towards a Holistic organisational Systems Model." Presented at 5th Int. Conf. Systems of Systems Engineering (SoSE), June 22-24, 2010, Loughborough, UK.
+
Rasmussen, R. E. (2007) The Wrong Target, MSc thesis, Joint Advanced Warfighting School, Norfolk, VA. Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/a468785.pdf
  
International Council on Systems Engineering (INCOSE), 2014, Systems Engineering Vision 2025 July, 2014; Accessed February 16 at http://www.incose.org/docs/default-source/aboutse/se-vision-2025.pdf?sfvrsn=4
+
Rittel, H.W.J., and M.M. Webber. 1973. "Dilemmas in a General Theory of Planning." Amsterdam, The Netherlands: Elsevier Scientific Publishing Company, Inc. p. 155–169, in Developments in Design Methodology, edited by N. Cross, 1984. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd. p. 135–144.  
  
Rittel, H.W.J., and M.M. Webber. 1973. "[[Dilemmas in a General Theory of Planning]]." Amsterdam, The Netherlands: Elsevier Scientific Publishing Company, Inc. p. 155–169, in ''Developments in Design Methodology,'' edited by N. Cross, 1984. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd. p. 135–144.
+
Sauser, B., Boardman, J., & Gorod, A. (2009) System of Systems Management, in System of Systems Engineering: Innovations for the 21st Century, M. Jamshidi (Ed.), (pp. 191–217) Wiley.
  
Tannenbaum, S.I., E. Salas, and J.A. Cannon-Bowers. 1996. "Promoting Team Effectiveness," in ''Handbook of Work Group Psychology'', edited by M.A. West. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd.
+
Siemieniuch, C.E. & Sinclair, M.A. (2014) Extending systems ergonomics thinking to accommodate the socio-technical issues of Systems of Systems, Appl. Ergon., V 45, Issue 1, Pages 85-98
  
Vernadat, F.B. 1996. ''Enterprise Modeling and Integration: Principles and Applications.'' London, England, UK: Chapman and Hall Publishers.
+
Van der Laar, P., Tretmans, J., & Borth, M. (2013) Situational Awareness with Systems of Systems. Springer.
  
 
===Primary References===
 
===Primary References===

Revision as of 15:45, 26 March 2017

Most systems of systems (SoS) are socio-technical systems that are composed of a number of interdependent resources, such as, people, processes, information, and technology that must interact with each other and their environment in support of a common mission (See also Enterprise Systems Engineering).

Managing the Socio-Technical Features of Systems of Systems

The managerial and operational independence of constituent systems of a SoS (Maier, 1998) imply that in general, SoS are socio-technical systems. Turning to views outside of Systems Engineering, Ergonomists regard socio-technical systems as having the following characteristics (Maguire, 2014):

  • There are collective operational tasks,
  • They contain social and technical sub-systems,
  • They are open systems (i.e. strongly interacting with their environments), and
  • The concept of the system being an unfinished system.

These are also characteristics of Systems of Systems. Klein (2014) has noted that approaches to socio-technical systems can take the two perspectives of “system affects people” or “people affect system”, depending upon how the system boundary is drawn. It is generally true for systems that consideration of their context requires socio-technical aspects to be taken into account.

Although there are many matters concerning the socio-technical aspects of SoS, there are two main issues, which are somewhat related, requiring attention. The first is the need for appropriate governance structures, given that operational and/or managerial independence affects top-down direction of the SoS and may compromise achievement of the SoS goal(s). The second issue is a lack of situational awareness of managers, operators, or other stakeholders of the SoS, so that they may not understand the impact of their local decisions on the wider SoS.

SoS Governance

Generally, design and operation of complex systems is concerned with control, but the classification of SoS (Dahmann, et. al., 2008) is based on the notion of diminishing central control, as the types go from directed to virtual. Sauser, et. al. (2009) has described the ‘control paradox of SoS’ and asserted that for SoS, ‘management’ is replaced by ‘governance’. Thus in C2 terms, the emphasis in SoS is on command, rather than control: ‘Control is a function of rules, time, and bandwidth; whereas command is a function of trusts, influence, fidelity, and agility’. Trusts, influence, and fidelity are intrinsically human qualities. The matters relating to the organizational aspects, such as these, are relevant to Enterprise Systems . Siemieniuch and Sinclair (2014) highlighted the complexity issues with respect to SoS by drawing attention to a variety of organizational characteristics that concern the interactions among system operators in a SoS:

Many agents, of different kinds Some degree of behavioral autonomy for agents Multiple steady states for agents Interactions between agents in an environment Lots of connections between agents Agents communicating in parallel Effects of an evolving environment Effects of evolving agents Interactions between different goals within an agent Interactions between agents with different goals Language/culture differences between agents

A major governance issue for SoS is understanding the ownership of, and making reliable estimates of risk (Fovino & Masera, 2007). High levels of connectivity, and the potential for emergent behavior due to the interactions of separately owned/operated constituent systems, means that significant risks may go unacknowledged and their mitigations unplanned.

In general, governance can be summed up by asking three connected questions (Siemieniuch and Sinclair, 2014):

  • Are we doing the right things (leadership)?
  • Are we doing those things right (management)?
  • How do we know this (metrics and measurements)?

Currently, there is no accepted framework for addressing these questions in a SoS context, but Henshaw et. al. (2013) have highlighted architectures as an important means through which governance may be clarified. They postulate that a SoS can be regarded as a set of trust and contract relationships between systems (i.e. including both informal and formal relationships). The systems architect of a constituent system must, therefore, address trust issues for each participating organization in the overall enterprise with which his/her system must interoperate. For SoS, technical engineering governance is concerned with defining and ensuring compliance with trust at the interface between constituent systems. An example of difficulty managing the interfaces in a SoS is provided in the Cassini-Huygens mission case study .

Situational Awareness

Situational awareness is a decision maker’s understanding of the environment in which he/she takes a decision; it concerns information, awareness, perception, and cognition. Endsley (1995) emphasizes that situational awareness is a state of knowledge. There are numerous examples of SoS failure due to the operator of one constituent system making decisions based on inadequate knowledge of the overall SoS (big picture). Fratricide incidents are among the most shocking (see for example Rasmussen, 2007).

On the other hand, SoS development is also viewed as the means through which improved situational awareness may be achieved (Van der Laar, et. al., 2013). In the defense environment, Network Enabled Capability (NEC) was a system of systems approach motivated by the objective of making better use of information sharing to achieve military objectives. Court (2005) identified that the NEC benefits chain was concerned with the quality of decision making, which relies on the quality of shared awareness, which relies, in turn, on networks and information quality; a crucial element is the sharing of information, knowledge, and understanding. NEC is predicated on the ability to share useful information effectively among the stakeholders that need it. It is concluded that improving situational awareness will improve SoS performance, or at least reduce the risk of failures at the SoS level. Thus, the principles which govern the organization of the SoS should support sharing information effectively across the network; in essence, ensuring that every level of the interoperability spectrum is adequately serviced. Operators need insight into the effect that their own local decisions may have on the changing SoS or environment; similarly they need to understand how external changes will affect the systems that they own.




References

Works Cited

Court, G. (2005) Validating the NEC benefits chain in 11th ICCRTS. Cambridge, UK. Retrieved from http://www.dodccrp.org/events/11th_ICCRTS/html/papers/155.pdf

Dahmann, J. S. & Baldwin, K. J. (2008) Understanding the Current State of US Defense Systems of Systems and the Implications for Systems Engineering, 2nd Annual IEEE Systems Conference, 1–7. http://doi.org/10.1109/SYSTEMS.2008.4518994

Endsley, M. R. (1995) Toward a Theory of Situation Awareness in Dynamic Systems, J. Human Factors and Ergonomics Soc., 37(1), 32–64. http://doi.org/10.1518/001872095779049543

Fovino, I. N., & Masera, M. (2007) Emergent disservices in interdependent systems and system-of-systems, in Proc. IEEE International Conference on Systems, Man and Cybernetics, Vol. 1, pp. 590–595. http://doi.org/10.1109/ICSMC.2006.384449

Henshaw, M. J. de C., Siemieniuch, C. E., & Sinclair, M. A. (2013) Technical and Engineering Governance in the Context of Systems of Systems, in NATO SCI Symp. Architecture Assessment for NEC (pp. 1–10). Tallinn, Es. NATO STO.

Henshaw, M. (2014) A Socio-Technical Perspective on SoSE, in Lecture Series in Systems of Systems Engineering for NATO Defence Applications (SCI-276). NATO CSO.

Klein, L. (2014) What do we actually mean by ‘sociotechnical’? On values, boundaries and the problems of language, Appl. Ergon., vol. 45, no. 2 PA, pp. 137–142.

Maguire, M. (2014) Socio-technical systems and interaction design - 21st century relevance, Appl. Ergon., vol. 45, no. 2 PA, pp. 162–170.

Rasmussen, R. E. (2007) The Wrong Target, MSc thesis, Joint Advanced Warfighting School, Norfolk, VA. Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/a468785.pdf

Rittel, H.W.J., and M.M. Webber. 1973. "Dilemmas in a General Theory of Planning." Amsterdam, The Netherlands: Elsevier Scientific Publishing Company, Inc. p. 155–169, in Developments in Design Methodology, edited by N. Cross, 1984. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd. p. 135–144.

Sauser, B., Boardman, J., & Gorod, A. (2009) System of Systems Management, in System of Systems Engineering: Innovations for the 21st Century, M. Jamshidi (Ed.), (pp. 191–217) Wiley.

Siemieniuch, C.E. & Sinclair, M.A. (2014) Extending systems ergonomics thinking to accommodate the socio-technical issues of Systems of Systems, Appl. Ergon., V 45, Issue 1, Pages 85-98

Van der Laar, P., Tretmans, J., & Borth, M. (2013) Situational Awareness with Systems of Systems. Springer.

Primary References

Checkland, P.B. 1981. Systems Thinking, Systems Practice. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd.

Hubbard, E-M., C.E. Siemieniuch, M.A. Sinclair, and A. Hodgson. 2010. "Working towards a Holistic Organisational Systems Model." Presented at 5th Int. Conf. Systems of Systems Engineering (SoSE), 22-24 June, 2010, Loughborough, UK.

Rittel, H.W.J., and Webber, M.M. 1973. "Dilemmas in a General Theory of Planning," in Policy Sciences 4. Amsterdam, The Netherlands: Elsevier Scientific Publishing Company, Inc. p. 155–169. In Cross, N. 1984. Ed. Developments in Design Methodology. Chichester, West Sussex, England, UK: John Wiley & Sons, Ltd. p. 135–144.

Additional References

Bruesburg, A., and G. Fletcher. 2009. The Human View Handbook for MODAF, draft version 2, second issue. Bristol, England, UK: Systems Engineering & Assessment Ltd. Available: http://www.hfidtc.com/research/process/reports/phase-2/hv-handbook-issue2-draft.pdf.

IFIP-IFAC Task Force. 1999. "The Generalised Enterprise Reference Architecture and Methodology," V1.6.3. Available: http://www.cit.gu.edu.au/~bernus/taskforce/geram/versions/geram1-6-3/v1.6.3.html.

ISO. 1998. ISO 14258:1998, Industrial automation systems — Concepts and rules for enterprise models. Geneva, Switzerland: International Organization for Standardization.

ISO. 2006. ISO 19439:2006, Enterprise integration — Framework for enterprise modelling. Geneva, Switzerland: International Organization for Standardization.

ISO. 2007. ISO 19440:2007, Enterprise integration — Constructs for enterprise modelling. Geneva, Switzerland: International Organization for Standardization.

Miller, F.P., A.F. Vandome, and J. McBrewster. 2009. Enterprise Modelling. Mauritius: Alphascript Publishing, VDM Verlag Dr. Müller GmbH & Co. KG.


< Previous Article | Parent Article | Next Article >


SEBoK v. 1.9.1 released 30 September 2018

SEBoK Discussion

Please provide your comments and feedback on the SEBoK below. You will need to log in to DISQUS using an existing account (e.g. Yahoo, Google, Facebook, Twitter, etc.) or create a DISQUS account. Simply type your comment in the text field below and DISQUS will guide you through the login or registration steps. Feedback will be archived and used for future updates to the SEBoK. If you provided a comment that is no longer listed, that comment has been adjudicated. You can view adjudication for comments submitted prior to SEBoK v. 1.0 at SEBoK Review and Adjudication. Later comments are addressed and changes are summarized in the Letter from the Editor and Acknowledgements and Release History.

If you would like to provide edits on this article, recommend new content, or make comments on the SEBoK as a whole, please see the SEBoK Sandbox.

blog comments powered by Disqus