“Snapshot” of the State of Software Reuse in Colombia

Defining the survey questions

In the first step, we defined the survey questions. These questions were based on the 26 factors found in the literature (Restrepo-Gutiérrez et al., 2021). The survey has 54 questions. It was built through an iterative process. Its final version was obtained after carrying out the preliminary evaluation, which is shown in Subsection 2.2. The survey has closed and open-ended questions distributed in three groups. Their distribution is presented in the scheme sheet (Restrepo-Gutiérrez, 2021):

• Survey main question: One survey question allowed us to investigate and measure the level of software reuse success and respond to the information collected through the other questions.

• Survey research questions: We defined 34 survey questions. Each of them was related to a research question in Table 1, which, in turn, was related to a factor. These responses and their relationship with the main question allow us to answer the 26 research questions.

• Survey complementary questions: We defined 19 questions to collect general information about the business and the person who takes the survey, namely company location, software application domain, number of employees, staff experience, and the person’s role. Therefore, some of these questions were used to collect information about company tools, framework usage, programming language usage, among others.

The survey was designed to be completed in less than 20 minutes; the whole questionnaire is available online (Restrepo-Gutiérrez, 2021).

Survey preliminary evaluation

In the second step, the survey questions were preliminarily evaluated by six software engineering research experts (internal review) and five software industry experts (external review).

Internal review: For the internal review, we asked the six research experts to evaluate each survey question according to a list of aspects to avoid in a survey, which have been defined byBarbara and Shari (2002)andLeung (2001). The list is available in the replication package byRestrepo-Gutiérrez (2021). We asked the research experts to send feedback via email; some changes were analyzed and applied to the survey questions based on it. This internal review resulted in minor changes such as clarifying terms and completing questions. Regarding the confusing terms, experts identified elements of the survey with unclear definitions that could result in misunderstandings. These elements were redefined for the sake of clarity. For example, one of the experts suggested defining the term systematic approach since it was not understood as referring to software quality models. Regarding the incomplete questions, experts suggested adding more items to the answer lists for some questions. For example, in the list of services offered by companies, hosting and software integration services should be added.

External review: After the internal review with research experts, we conducted a review with software industry experts. This evaluation included people with a high degree of experience in the field (head of the technology department, product manager, software innovation leader, and business analyst roles). These people were associated with well-known software companies in Colombia, located in several departments of Colombia. To evaluate the survey, we held a one-hour meeting with each industry expert. We presented the contents of the content in each meeting, and we explained the research goal to each industry expert. We also requested that each industry expert complete the survey aloud: “say everything you think" (GAO, 2009). Then, we asked them to find issues regarding the survey questions. We recorded and reviewed these issues. We also discussed the survey structure (topic coverage, survey length, sensitivity, and missing questions) with the industry experts. The final survey version was finished and shared with the external reviewers.

Identifying the survey target population

The survey target population was companies in Colombia, whose main economic activity is software development. To identify this population, we got information from the Colombian Chamber of Commerce. This entity has a complete database that includes information that allows identifying the population under study. We got information of companies established in Colombia grouped by type of economic activity. Using the database, we got 5.818 companies whose main economic activity was software development. The filters used are available in a replication package (Restrepo-Gutiérrez, 2021). From these 5.818 software companies, a sample of 361 was determined for the survey, considering a confidence level of 95% and a margin error of 5% according to the general rule regarding acceptable margins of error mentioned by Kotrlik and Higgins (2001). Since the response rates are typically well below 100%, the authors recommended oversampling: “if you are mailing out surveys or questionnaires, count on increasing your sample size by 40%-50% to account for lost mail and non-responders” (Salkind, 2017, p. 107). For this reason, oversampling was used to anticipate the response rate. To calculate the response rate, the results from the first week after sending the survey were used to estimate how many additional responses were required. As the response rate was 9% of the initial sample of 361 companies, it was established that the survey would be randomly sent to 4.011 companies.

Survey data collection

We defined a standardized email to send the survey to the target population. We applied the following strategy to send the standardized email: (i) if the company provided an email address (for example, on its website), we sent the standardized email to that email address; or (ii), if the company did not provide an email address, we tried to contact it through the Contact us section, phone calls, or social networks (Facebook, Messenger, LinkedIn, and WhatsApp Business). 2.534 software companies were contacted through email and 1.477 by other means. To motivate the respondents to share their experiences freely, it was mentioned that their answers would remain anonymous and that the results would be shared when they were published. It is important to note that we internally recorded each company’s name and location (department). This information was used to guarantee that the survey was answered by no more than one representative per company. To manage the survey responses, we used an online survey platform. In the end, 367 companies fully completed the online survey, thus complying with the sample needed, with an average time responding of 15 minutes and a withdrawing rate of 37%.

Survey data analysis

This project was based on quantitative survey data, allowing us to perform statistical analyses to test our research questions. Descriptive statistics and multinomial and ordinal logistic regressions were used to analyze the research questions. This analysis aimed at determining which of the responses of the survey research questions (predictor variables) were the most influential over the survey main question responses (response variable), and, in the end, to be able to answer the research questions. Logistic regression models describe the relationship between a response variable and one or more explanatory variables. The strength of logistic regression models is their ability to handle many variables, some of which may be on different measurement scales (Harrell, 2015). There are different logistic regression models: multiple, multinomial, and ordinal logistic. Multiple logistic regression models consider more than one independent variable, ordinal logistic regression models consider the ordinal nature of the outcome, and multinomial logistic regression handles the case where the outcome variable is nominal with more than two levels (Harrell, 2015). As there is only one independent variable in the study, multiple logistic regression models were ruled out. Instead, multinomial and ordinal logistic regression models were chosen to describe the relationship between variables.

The response variable (survey main question responses) was an ordinal numerical scale from 1 to 10, where 1 was unsuccessful, and 10 was very successful. This scale was grouped by two values, thus obtaining a scale of 5 levels from no success to strong success (no success, limited success, moderate success, substantial success, and strong success), where the values 1 and 2 of the initial scale correspond to the no success level and so on. Some of the predictor variables (survey research question responses) are also ordinal, and they go from strongly disagree to strongly agree (questions with the Likert answer scale). Other predictor variables were defined as categorical. The main question (related to the response variable) and an example of a survey research question (related to a predictor variable) can be found in the aforementioned replication package (Restrepo-Gutiérrez, 2021). The type of variable for each survey question can be viewed in the scheme sheet (Restrepo-Gutiérrez, 2021). To analyze open-ended questions, each answer was hand-coded into one of several categories defined as presented on the sheet questions (Restrepo-Gutiérrez, 2021). These categories were created with a general review of the answers.

It should be noted that multiple-choice questions cannot be considered as single variables, so each item of those questions counted as an independent variable. Therefore, there were 135 independent variables and one dependent variable in each model (Restrepo-Gutiérrez, 2021). Although some variables may be correlated, this analysis is not contemplated within the scope of this article. Each model required the response variable and the predictor variables. For the multinomial logistic regression, the response variable was entered as nominal. The response variable was ordinal for the ordinal logistic regression. The predictor variables associated with Likert-scale answers could be entered as nominal or ordinal. For this reason, four models were created with the variation resulting from taking the response and predictor variables as ordinal and nominal, respectively. In order to create the models, the polr and multinom commands from the MASS package of the R tool were used (Ripley, 2020). To estimate the best logistic regression model, an iterative model evaluation was performed with the Step command from the R stats package to fit the model. Step is an automated method based on AIC (Akaike Information Criterion) model selection that aims to find out the best fit model from all possible subset models and returns the optimal set of features (Hosmer et al., 2013). The four models were evaluated with the AIC. The replication package (Restrepo-Gutiérrez, 2021) has the created and evaluated models. As a result of this process, models 3 and 4 had the lowest AIC and the same value. The lowest AIC between models 1 and 2 was considered to decide on the model to choose, thus resulting in model 1, where the predictor variable is nominal. model 4 was also chosen, where the response variable is ordinal, and the predictor variables are nominal. The results of the regression are available in the referenced literature (Restrepo-Gutiérrez, 2021).

Threats to validity in the survey

Threats to the survey’s validity can be analyzed from the points of view of construct, internal, external, and statistical conclusion validity.

Construct validity: Construct validity addresses how well whatever is purported to be measured has been actually measured (Lavrakas, 2008). Verifying survey questions for construct validity is essential, especially when these questions are self-developed. We followed three steps to verify the construct validity. First, we performed a series of validation tasks for identifying and calibrating ambiguously worded surveys. Second, four experienced researchers not involved in the questionnaire design were consulted to discuss each question. Then, the survey was revised based on the feedback collected to ensure that the closed questions were easy to interpret and sufficiently complete. Finally, a pretest was conducted to finetune the survey; it was taken by two researchers. They were asked to comment on their understanding of each item after filling out the questionnaire. Based on their feedback, minor revisions were performed.

Internal validity: Internal validity considers whether the experimental design can support conclusions on causality or correlations (Morisio et al., 2002). Questionnaires should have items that are logically interrelated to elicit information about the constructs under analysis without defects that would distort the information. The questionnaire presented in Restrepo-Gutiérrez (2021) was evaluated for internal consistency by computing Cronbach’s alpha test, which is the most widely used measure of reliability. It describes the internal validity (Hinkin, 1998), which is how much each measured element correlates to each other, or the overall consistency of the questionnaire (Cronbach, 1951). According to Hinkin (1998), an acceptable value for Cronbach alpha is 0,7, but a minimum of 0,8 is the most desirable value. The evaluated questionnaire has a Cronbach’s alpha of 0,9. Therefore, the questionnaire’s internal coherence is acceptable, and the results presented in Section 3 are valid.

The accuracy of the answers is difficult to calculate due to the anonymity of the respondents (Singh, 2011). To address this issue, no incentives were given for responding to the survey other than the self-interest of respondents to complete it. The answers were also reviewed. As a result of this process, 222 incomplete and 6 completed surveys were rejected due to inconsistency in the data, such as company name not within our list of software companies or meaningless data in open-ended questions. Additionally, the impact on the data depends on the size of the group surveyed. In the research, we have 367 reviewed and completed surveys. It would take several people entering the wrong information to make a noticeable difference. In this case, the use of a survey protects us against bad actors. The dates of the studies found in the literature are different, so the comparisons are approximate since the state may have changed over time. The only way to avoid this threat to the validity of the comparison is to conduct the study in parallel. Such a study is costly, and we do not have the resources to do it.

External validity: It refers to the extent to which the research findings based on a sample of individuals or objects can be generalized to the same population from which the sample is taken or to other similar populations in terms of contexts, individuals, times, and settings (Lavrakas, 2008). The defined sample represents the population in our survey, and the participants were chosen using a random sampling method. Therefore, our research can be generalized to describe software organizations in Colombia. Also, to avoid survey non-response effects, the response rate was maximized by sending reminder emails and re-sending the uncompleted survey.

Statistical conclusion validity: It refers to the accuracy of a conclusion regarding a relation between or among variables of interest. As the sample size is considerable, there is no bias in the assumptions made of the descriptive statistics. Nevertheless, due to the large number of variables in the logistic regression model, there may be a bias in the results given by model 4. For this reason, a chi-squared-type statistic to test the validity of the logistic regression model was applied, using the anova command from the stats package of R (Harrell, 2015; Hastie and Pregibon, 2017), which compares model 4 before executing the Step command against the resulting model. The residual deviance of the model 4 was 521,8, 477 for the resulting model, and the Pr(Chi) was 0,9998, nearly 1, which indicates that the resulting model is better than model 4 and concludes that the resulting predictor variables have significant interactions with the independent variable. Another threat to the statistical conclusion validity is that the survey results focused on the impact of individual factors on the dependent variable, but some influences may imply combinations of factors. Finally, and as it was not contemplated, a correlation analysis within this article’s scope is considered a threat that will be mitigated in future work.